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PREFACE 


This  study  was  conducted  within  RAND’s  Center  for  the  Study  of  Food  and  Nutrition 
Policy  as  part  of  RAND’s  Child  Nutrition  Analysis  Project  (CNAP)  with  the  United 
States  Department  of  Agriculture’s  Food  and  Nutrition  Service  under  “Obtaining 
States’  Nutritional  Monitoring  Data.”  This  document  reports  on  a  study  of  state  nu¬ 
tritional  monitoring  data  and  should  be  of  benefit  to  those  interested  in  the  opera¬ 
tion  of  child  nutrition  programs. 

This  volume  contains  the  narrative  report.  An  appendix  volume  (Liisa  Fliatt  and 
Jacob  Alex  Klerman,  State  Monitoring  of  National  School  Lunch  Program  (NSLP) 
Nutritional  Content:  State -by-State  Results,  RAND,  MR-1296/ 1-USDAFNS,  2001) 
provides  the  detailed  state  responses. 

The  main  text  of  this  report  was  written  in  the  fall  of  1999.  Since  that  time,  there  have 
been  some  changes — for  example,  the  Dietary  Guidelines  for  Americans  were  up¬ 
dated  in  2000.  The  text  of  this  report  mentions  only  the  1995  update  because  the 
2000  update  was  not  completed  when  the  report  was  written. 
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SUMMARY 


INTRODUCTION 

As  part  of  the  School  Meals  Initiative  (SMI),  the  U.S.  Department  of  Agriculture-Food 
and  Nutrition  Service  (USDA-FNS)  now  requires  each  state  to  regularly  review  the 
nutritional  content  of  food  served  by  each  School  Food  Authority  (SFA)  as  part  of  the 
National  School  Lunch  Program  (NSLP).  While  states  must  monitor  the  nutritional 
content  of  school  meals,  they  are  not  required  to  forward  any  information  to  USDA- 
FNS.  However,  USDA-FNS  is  required  to  measure  progress  against  Objective  2.1  of 
FSN’s  1997-2002  Strategic  Plan,  which  states  that  it  will  “ensure  tliat  school  meals  are 
consistent  with  the  Dietary  Guidelines  for  Americans  [DGA]  and  the  Recommended 
Daily  Allowances  [RDA].”  In  September  2000,  FNS  issued  a  completely  revised 
Strategic  Plan,  which  established  goals  for  school  lunches  under  a  new  Objective  1.3: 
"Improved  nutritional  quality  of  meals,  commodities,  and  other  program  benefits.” 
The  “target”  established  under  Objective  1.3  is  “By  2005,  reach  less  than  or  equal  to 
30%  calories  from  total  fat  and  less  than  10%  calories  from  saturated  fat;  maintain 
calorie,  vitamin  and  mineral  content  at  greater  than  or  equal  to  33%  of  RDA.”  To 
measure  progress,  USDA-FNS  needs  to  produce  state  and  national  aggregations  of 
the  nutrient  content  in  school  lunches  to  show  that  meals  in  a  given  state  or  in  the 
country  as  a  whole  are  consistent  with  the  DGA  and  RDA.  In  addition,  FNS  needs  to 
do  this  while  imposing  minimal  reporting  burdens  on  the  states. 

The  USDA-FNS  contracted  with  RAND  to  look  more  carefully  at  how  some  states  are 
performing  SMI  reviews  and  how  they  might  be  able  to  transmit  data  with  limited 
additional  burden.  In  particular,  it  asked  seven  states  to  work  collaboratively  with 
RAND — through  on-site,  in-depth  interviews — to  explore  possible  reporting  systems: 
California,  Georgia,  Nebraska,  New  Jersey,  New  York,  Texas,  and  Wisconsin.  These 
states  were  chosen  as  representative  of  regional  differences  across  the  country  and  of 
methods  for  conducting  SMI  reviews.  This  report  describes  the  results  of  the  analysis 
of  possible  approaches  for  states  to  report  the  results  of  their  nutritional  reviews  to 
USDA-FNS. 

WHAT  DATA  SHOULD  BE  PROVIDED? 

To  design  an  appropriate  system,  USDA-FNS  must  determine  what  information  from 
the  SMI  reviews  it  needs,  while  again  minimizing  the  burden  to  the  states.  Starting 
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with  an  exhaustive  list  of  data  elements  to  collect,  we  worked  with  the  selected  states 
to  identify  a  set  of  required  elements;  we  also  identified  some  optional  elements  and 
some  elements  that  should  not  be  included.  The  three  sets  are  summarized  in  Table 
S.l. 

HOW  DO  STATES  CURRENTLY  PERFORM  SMI  REVIEWS? 

To  understand  each  state’s  ability  to  report  the  required  information  and  to  develop 
a  system  that  would  impose  the  minimum  burden  on  the  states,  we  examined  how 
the  states  currently  perform  SMI  reviews.  Overall,  the  seven  states  interviewed  fol¬ 
lowed  a  similar  process  in  which  the  SFA  gathers  information  for  review,  the  nutrient 
analysis  is  performed,  a  correction  plan  is  developed  with  the  SFA,  and  the  review  is 
completed.  However,  each  state  operates  slightly  differently.  Table  S.2  summarizes 
the  information  collected  on  how  SMI  reviews  are  conducted  in  the  seven  states, 
showing  the  variation.  Whether  the  state  office  receives  the  data  and  whether  the 
state  has  compiled  it  into  any  type  of  central  database  are  particularly  pertinent. 
Except  for  Texas,  all  the  states  receive  the  data,  and  most  have  databases. 


HOW  SHOULD  THE  DATA  BE  COLLECTED? 

Although  there  are  a  number  of  options  for  collecting  the  data,  given  USDA-FNS’s 
stated  goal  of  collecting  data  at  minimal  cost  to  the  states,  it  is  appropriate  for  the 
USDA-FNS  to  accept  the  data  in  any  reasonable  format  the  states  are  willing  to  pro¬ 
vide,  including  USDA  file  formats,  other  file  formats  from  states  (assuming  the  for¬ 
mats  contain  all  the  required  data  elements),  and  through  an  Internet  form.  USDA- 
FNS  should  also  consider  ensuring  that  whatever  format  is  used,  the  states  will  be 
able  to  customize  the  databases  for  their  own  purposes.  Finally,  given  the  five-year 
SMI  review  cycle,  USDA-FNS  will  need  only  a  moderate-sized  database,  which  can  be 
run  on  the  standard  desktop  personal  computer  purchased  today  (late  1999).  The 

Table  S.l 

Summary  of  Data  Elements  to  Be  Required,  Optional,  and  Not  Included 


Required  Elements 

Optional  Elements 

Elements  Not 
Included 

Type  of  SFA 

SFA  identifier 

Name  of  reviewer 

Menu  planning  system  used 

Review  week 

Dates  of  review 

Analysis  lunch  only,  breakfast  only, 
or  lunch  and  breakfast 

Contact  person 

Standards 

Analysis  weighted? 

Classification  of  “any 
reasonable  approach” 

Review  done  by  grade  or  age 

Urban  or  rural  area 

Lowest  grade/age 

Size  of  district 

Highest  grade/age 

Estimated  average  daily  participation 
Actual  output  from  analysis  software 

Comments 

Summary  xiii 


Table  S.2 

Summary  of  How  SMI  Reviews  Are  Performed  in  the  Seven  States 


State 

Who  Does  Analysis 

Reviewer  Other 

Weighted 

Analysis? 

State  Has 

Data?* 

Software 

Used 

California 

X 

X 

Some 

Y,  DB 

NutriKids 

Georgia 

X 

X 

Y 

Y,  NDB 

NutriKids 

Nebraska 

X 

Y 

Y,  DB 

NutriKids 

New  Jersey 

X 

Y 

Y,  NDB 

NutriKids 

New  York 

X 

X 

Y 

Y,  DB 

NutriKids 

Texas 

X 

N 

N 

NutriKids 

Wisconsin 

X 

X 

Y 

Y,  DB 

NutriKids 

*08  =  database  or  spreadsheet;  NDB  =  no  database  or  spreadsheet 


tabulations  required  are  straightforward  and  could  be  precoded.  An  analyst  with 
rudimentary  experience  in  statistical  analysis  should  be  able  to  program  the  required 
queries  without  major  assistance. 

DISCUSSION  OF  THE  CONTENT  OF  THE  SMI  EFFORT 

We  also  considered  the  SMI  effort  more  broadly.  We  found  some  positive  effects  of 
the  current  review  procedures.  The  experience  of  preparing  for  a  review  itself  raises 
consciousness  of  the  nutritional  issues,  and  the  results  of  the  review  lead  to  im¬ 
provements  in  the  nutritional  content  of  meals. 

We  also  found  three  issues  about  the  quality  of  the  data  collected.  First,  states  have 
concerns  about  inter-rater  reliability;  two  reviews  of  the  same  meals  might  reach  dif¬ 
ferent  conclusions.  Although  such  problems  are  nearly  impossible  to  eliminate,  they 
can  be  minimized  by  promulgating  common  procedures  for  reviews,  providing  for¬ 
mal  initial  and  ongoing  training  in  those  procedures,  and  initiating  a  program  to  test 
the  quality  of  completed  reviews. 

Second,  even  if  the  reviews  are  perfect,  are  they  representative  of  meals  at  the  SFA 
level?  In  other  words,  how  much  information  do  the  results  of  a  review  (of  meals 
served  one  given  week  in  a  given  school)  yield  about  all  the  meals  served  in  that 
school  (or  SFA)?  To  address  this  issue,  USDA-FNS  could  fund  a  small  pilot  study  that 
would  randomly  select  a  small  group  of  SFAs  and  collect  SMI  type  information  across 
schools  and  through  time. 

Third,  are  the  reviews  at  the  state  level  representative?  While  the  USDA  Strategic 
Plan  requires  information  about  all  meals  served,  review  processes  can  only  collect 
information  about  meals  reviewed.  Standard  arguments  from  statistical  sampling 
imply  that  it  is  possible  to  construct  a  statistically  accurate  portrait  of  all  meals 
served  by  a  group  of  SFAs  (e.g.,  a  state)  by  averaging  results  across  multiple  reviews 
and  across  individual  weeks  from  individual  schools  from  multiple  districts.  For 
these  arguments  to  apply,  reviewed  meals  must  be  selected  randomly,  but  the  cur¬ 
rent  procedures  do  not  guarantee  random  selection. 
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In  addition,  more  precise  estimates  could  be  obtained  with  a  different  review  cycle. 
Currently,  one  review  is  required  per  SFA  in  each  five-year  cycle,  regardless  of  district 
size.  However,  SFAs  vary  in  size  dramatically.  For  example,  about  half  the  meals  in 
New  York  State  are  served  by  the  New  York  City  public  school  system.  Statistical 
considerations  suggest  allocating  more  resources  to  large  SFAs  and  fewer  resources 
to  smaller  SFAs.^  Such  a  procedure  would  also  ease  the  burden  on  smaller  SFAs, 
probably  reducing  opposition  to  the  SMI  effort. 


^Effective  allocation  of  the  sample  to  minimize  variance  across  states  as  a  whole  requires  more  resources 
for  larger  SFAs. 
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Chapter  One 

INTRODUCTION 


As  part  of  the  School  Meals  Initiative,  the  U.S.  Department  of  Agriculture-Food  and 
Nutrition  Service  (USDA-FNS)  now  requires  each  state  to  regularly  review  the  nutri¬ 
tional  content  of  food  served  by  each  School  Food  Authority  (SFA)^  as  part  of  the 
National  School  Lunch  Program  (NSLP).  However,  the  data  collected  as  part  of  this 
system  of  reviewing  nutritional  content  are  not  currently  reported  to  anyone  outside 
of  the  state  in  which  they  are  collected.^  This  report  describes  the  results  of  a  RAND 
analysis  of  possible  approaches  for  states  to  report  the  results  of  their  nutritional  re¬ 
views  to  USDA. 


POLICY  CONTEXT 

In  this  section,  we  briefly  review  the  history  of  the  National  School  Lunch  Program, 
the  School  Meals  Initiative  (SMI),  and  SMI  reviews. 


USDA-FNS  School  Nutrition  Programs 

The  NSLP  was  estabhshed  in  1946  by  the  National  School  Lunch  Act.  The  legislation’s 
preamble  stated  its  goals: 

as  a  measure  of  national  security,  to  safeguard  the  health  and  well-being  of  the 
Nation’s  children  and  to  encourage  the  domestic  consumption  of  nutritious  agricul¬ 
tural  commodities  and  other  food,  by  assisting  the  States,  through  grants-in-aid  and 
other  means,  in  providing  an  adequate  supply  of  foods  and  other  facilities  for  the  es¬ 
tablishment,  maintenance,  operation,  and  expansion  of  nonprofit  school  lunch  pro¬ 
grams.^ 

Through  subsequent  amendments  and  increased  funding,  the  program  has  ex¬ 
panded  to  95,000  public  and  private  schools  and  residential  child  care  institutions. 
More  than  26  million  children  receive  free  or  reduced-price  lunches  every  day. 
Roughly  92  percent  of  schoolchildren  in  the  United  States  are  in  schools  with  lunch 


^An  SFA  is  usually  a  public  school  disUict,  a  private  school,  a  charter  school,  a  residential  child  care 
institution  (RCCl),  or  a  juvenile  detention  center. 

^At  the  time  this  report  was  written,  FNS  did  not  require  any  data  from  the  states.  Subsequendy,  some  of 
the  regional  offices  of  FNS  have  required  states  to  submit  these  data  quarterly.  Specifically,  New  York 
reported  to  us  that  the  Northeast  Region  requires  quarterly  submissions. 

^Section  2  of  the  National  School  Lunch  Act,  42  USC  1751. 
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programs,  and  about  58  percent  participate  in  the  program  on  an  average  school  day. 
For  fiscal  year  1998,  the  program  had  a  budget  of  $4.2  billion.^ 

While  the  NSLP  was  intended  to  “safeguard  the  health  and  well-being  of  the  Nation’s 
children,”^  through  1996  the  program’s  nutritional  focus  was  on  broad  food  groups 
and  total  calories.  In  those  fifty  years,  the  nutritional  status  of  Americans  and  scien¬ 
tific  understanding  of  proper  nutrition  both  changed.  Those  changes  were  reflected 
in  a  series  of  Dietary  Guidelines  for  Americans  (DGA)  beginning  in  1980  and  most  re¬ 
cently  in  1995.®  The  1990  DGA  and  the  Food  Pyramid  that  was  developed  to  publi¬ 
cize  them  reflected  significant  changes  in  the  content  of  the  guidelines.  The  lan¬ 
guage  of  the  new  guidelines  was  more  positive,  was  oriented  toward  the  total  diet, 
and  provided  more  specific  information  regarding  food  selection.  For  the  first  time, 
numerical  recommendations  were  made  for  intakes  of  dietary  fat  and  saturated  fat.^ 
These  changes  were  further  refined  in  the  1995  DGA. 

Concern  about  the  content  of  school  meals  and  their  role  in  promoting  the  nutrition 
of  children  was  further  focused  by  the  1992  School  Nutrition  Dietary  Assessment 
(SNDA).  It  found  that  many  school  meals  did  not  meet  the  then-current  1990  DGA. 
The  study  showed  that  the  average  school  lunch  had  27  percent  more  calories  from 
fat,  twice  as  much  sodium,  and  50  percent  more  calories  from  saturated  fat  than  the 
recommendations.®  In  addition,  SNDA  found  that  children  who  ate  the  hot  school 
lunch  were  consuming  significantly  higher  numbers  of  calories  from  fat  than  chil¬ 
dren  who  obtained  their  lunches  from  other  sources.® 

SCHOOL  MEALS  INITIATIVE  AND  THE  PROPOSED  RULE 

In  response,  public  hearings  were  held  and  USDA  created  a  plan  for  the  NSLP  that 
ultimately  became  the  School  Meals  Initiative  (SMI).  From  the  public  comments, 
USDA  developed  five  Guiding  Principles  for  SMI: 

•  Healthy  Children:  Provide  school  meals  that  promote  health,  prevent  disease, 
and  meet  the  Dietary  Guidelines  for  Americans  (DGA). 

•  Customer  Appeal:  Serve  meals  that  children  will  want  to  eat. 

•  Flexibility:  Reduce  paperwork,  streamline  reporting  systems,  recognize  differ¬ 
ences,  and  offer  different  methods  for  meeting  NSLP  guidelines. 

•  Investing  in  People:  Provide  technical  assistance  to  schools  so  that  they  can  im¬ 
prove  the  nutritional  content  of  school  meals. 


^From  http; //www.fns. usda.gov/cnd/Lunch/AboutLunch/faqs.htm. 

^Section  2  of  the  National  School  Lunch  Act,  42  USC  1751. 

®When  this  report  was  written,  the  1995  DGA  were  the  most  recent.  The  2000  version  has  since  come  out 
but  is  not  referenced  in  this  report. 

^Dietary  Guidelines  Advisory  Committee,  1995,  Appendix  1:  History  of  Dietary  Guidelines  for  Americans. 
®Burghardt,  et  al.,  1993. 

®60  FR  31195,  June  13,  1995. 
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•  Building  Partnerships:  Create  partnerships  with  public  and  private  entities  to 
ensure  continuing  collaborative  efforts. 

On  June  10,  1994,  USDA  published  a  proposed  rule  that  updated  and  expanded  the 
nutrition  standards  for  school  meals  and  required  them  to  meet  the  DGA.  In  particu¬ 
lar,  school  meals  were  to  satisfy  quantitative  standards  for  percentage  of  calories 
from  fat,  percentage  of  calories  from  saturated  fat,  protein  (g),  calcium  (mg),  iron 
(mg),  vitamin  A  (RE),  vitamin  C  (mg),  and  calories  (kcal).  In  addition,  qualitative 
goals  stated  that  school  meals  should  show  a  reduction  in  the  level  of  cholesterol  and 
sodium  and  an  increase  in  the  level  of  fiber.  Since  the  rule  gives  no  quantitative 
standards  with  which  to  measure  these  changes,  states  were  to  be  allowed  to  achieve 
these  guidelines  in  one  of  two  ways.  They  could  choose  a  quantitative  standard  for 
these  nutrients  and  show  how  the  nutrient  content  of  their  school  meals  compares  to 
this  standard  or  they  could  choose  to  use  an  SFA’s  initial  nutrient  analysis  as  a  base¬ 
line  and  show  the  change  in  subsequent  nutrient  analyses.  See  Table  1.1  for  the  cur¬ 
rent  standards  by  grade  range. 

To  assure  that  school  meals  served  satisfied  the  DGA,  the  proposed  rule  would  have 
replaced  the  NSLP’s  rigid  food-based  menu  planning  system  that  had  been  in  place 
since  the  beginning  of  the  program  with  a  menu  planning  system  based  on  comput¬ 
erized  nutrient  analysis.  In  particular,  the  Nutrient  Standard  Menu  Planning 
(NuMenus)  system  would  have  required  SFAs  to  perform  a  nutrient  analysis,  using 
USDA-approved  computer  software,  on  all  foods  offered  as  part  of  a  reimbursable 
meal  in  the  meals  programs  prior  to  the  service  of  the  meals. 

The  computerized  nutrient  analysis  would  have  provided  the  SFA  with  a  nutrient 
analysis  of  each  proposed  menu.  Given  that  analysis,  it  would  have  been  straight¬ 
forward  to  verify  that  the  proposed  menu  was  in  compliance  with  the  DGA,  and  thus 
with  the  requirements  of  the  NSLP  authorizing  legislation.  If  trial  menus 

Table  1.1 

Nutrient  Standards — NuMenus  and  Assisted  NuMenus 


Nutrient 

Standards  by  Grade  Range 

Preschool 

K-6 

7-12 

Optional  K-3 

Calories  (kcal) 

>517 

>664 

>825 

>633 

Total  Fat  (as  %  of  food 

<30% 

<30% 

<30% 

<30% 

energy) 

Saturated  Fat  (as  %  of 

<10% 

<10% 

<10% 

>10% 

food  energy) 

Protein  (g) 

>7 

>10 

>16 

>9 

Calcium  (mg) 

>267 

>286 

>400 

>267 

Iron  (mg) 

>3.3 

>3.5 

>4.5 

>3.3 

Vitamin  A  (RE) 

>150 

>224 

>300 

>200 

Vitamin  C  (mg) 

>14 

>15 

>18 

>15 

Cholesterol  (mg) 

A  reduction  in  level  of  cholesterol 

Sodium  (mg) 

A  reduction  in  the  level  of  sodium 

Fiber  (g) 

An  increase  in  the  level  of  fiber 
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were  not  in  compliance,  SFAs  could  iteratively  make  adjustments  to  the  trial  menu 
until  it  was  in  compliance. 

The  proposed  rule  recognized  that  not  all  SFAs  would  have  the  capability  to  do  nutri¬ 
ent  analysis  and  therefore  provided  for  a  variation  on  this  system.  That  variation, 
called  Assisted  NuMenus,  allowed  the  SFA  to  have  a  third  party  perform  the  nutrient 
analysis.  Finally,  the  proposed  regulation  required  the  state  agencies  to  assess  the 
nutrient  analyses  being  performed  by  the  SFAs  and  to  take  appropriate  actions  if  ei¬ 
ther  the  analyses  were  not  being  performed  correctly  or  the  meals  were  not  in  com¬ 
pliance  with  the  DGA. 

Thus,  through  NuMenus  or  Assisted  NuMenus,  the  proposed  rule  included  a  mecha¬ 
nism  through  which  school  meals  could  plausibly  be  expected  to  be  in  compliance 
with  the  DGA  by  the  1998-1999  school  year.  However,  before  the  rule  could  be  final¬ 
ized,  PL  103-448,  the  Healthy  Meals  for  Healthy  Americans  Act  of  1994,  was  passed. 
Responding  to  widespread  complaints  about  the  burden  of  nutrient  analysis,  the 
legislation  instructed  USDA  to  allow  SFAs  to  continue  using  a  food-based  system  of 
menu  planning,  in  addition  to  NuMenus  or  Assisted  NuMenus.  The  legislation  also 
required  that  school  meals  be  in  compliance  with  the  nutrient  requirements  by  the 
1996-1997  school  year  (two  years  earlier  than  under  the  proposed  rule).  However, 
since  the  legislation  allowed  menu  planning  systems  that  did  not  automatically  in¬ 
volve  nutrient  analysis,  there  was  no  direct  mechanism  through  which  food-based 
SFAs  could  be  expected  to  verify  that  their  meals  were  indeed  in  compliance  with  the 
DGA. 

On  January  27,  1995,  USDA  published  a  revised  proposed  rule.  It  supplemented  the 
June  10,  1994  proposed  rule.  It  incorporated  the  shorter  timeline  and  the  inclusion 
of  food-based  menu  planning  systems  from  PL  103-448.  In  the  January  rule,  USDA 
proposed  a  food-based  system,  called  Enhanced  Food-Based  menu  planning,  that 
was  very  similar  to  the  previous  food-based  system.  The  only  difference  from  a  tra¬ 
ditional  food-based  system  was  the  inclusion  of  more  fruits  and  vegetables  and 
grains  and  breads.  To  implement  the  requirement  that  NLSP  meals  meet  the  DGA, 
but  in  the  absence  of  a  requirement  that  every  SFA  perform  a  computerized  nutrient 
analysis  for  every  menu,  the  revised  proposed  regulation  required  each  State  Agency 
to  monitor  those  SFAs  using  food-based  systems.  That  monitoring  was  to  consist  of  a 
nutrient  analysis  of  each  SFA  at  least  every  five  years. 

THE  FINAL  RULE  AND  THE  HEALTHY  MEALS  FOR  CHILDREN  ACT 

The  final  rule,  which  was  published  June  13,  1995,  in  the  Federal  Register  and  was 
based  on  the  proposed  rules  from  June  1994  and  January  1995.  In  May  1996, 
Congress  enacted  the  Healthy  Meals  for  Children  Act  (PL  104-149),  which  added  two 
more  menu  planning  systems — the  system  that  had  been  in  place  since  1946 
(Traditional  Food-Based)  and  another  called  “Any  Reasonable  Approach.”  Under  the 
latter  option,  states  could  develop  their  own  menu  planning  system  as  long  as  it  met 
the  requirements  laid  out  by  FNS  for  school  meals  and  nutrition. 
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Since  the  final  regulations  allowed  for  five  different  menu  planning  systems  (see 
Table  1.2),  several  of  which  did  not  include  having  the  SFA  perform  nutrient  analysis, 
it  required  much  more  work  for  the  State  Agency  than  the  June  1994  version  of  the 
rule  would  have.  Under  the  original  proposed  rule,  all  SFAs  would  have  been  re¬ 
quired  to  perform  their  own  nutrient  analyses  or  hire  someone  to  do  the  analyses  be¬ 
cause  they  would  have  had  to  use  NuMenus  or  Assisted  NuMenus.  While  the  final 
rule  and  the  mandates  under  the  Healthy  Meals  for  Children  Act  allowed  the  SFAs 
more  flexibility,  they  also  gave  the  SFAs  the  opportunity  to  choose  menu  planning 
systems  that  did  not  require  them  to  perform  their  own  analyses.  Therefore,  the 
states  were  required  to  put  a  system  in  place  that  would  allow  state  staff  or 
contractors  to  perform  the  analyses. 

Under  the  current  SMI  regulations,  states  are  required  to  monitor  the  nutritional 
content  of  school  meals,  but  they  are  not  required  to  forward  any  information  to 
USDA — either  about  which  reviews  have  been  completed  or  about  the  nutritional 
content  of  the  meals  reviewed  or  about  the  improvement  plans  developed.  FNS, 
however,  is  required  to  measure  progress  against  Objective  2.1  of  FNS’s  1997-2002 
Strategic  Plan,  which  states  that  FNS  will  “ensure  that  school  meals  are  consistent 
with  the  Dietary  Guidelines  for  Americans  and  the  Recommended  Daily  Allowances.” 
The  “target”  established  under  Objective  1.3  is  “By  2005,  reach  less  than  or  equal  to 
30%  calories  from  total  fat  and  less  than  10%  calories  from  saturated  fat;  maintain 
calorie,  vitamin  and  mineral  content  at  greater  than  or  equal  to  33%  of  RDA.”  To  do 
this,  FNS  needs  to  produce  state  and  national  aggregations  of  the  nutrient  content  in 
school  lunches,  so  that  FNS  can  show  that  meals  in  a  given  state  or  in  the  country  as 
a  whole  are  meeting  the  given  objective  of  ensuring  that  school  meals  are  consistent 
with  the  Dietary 

Guidelines  and  the  RDAs.  In  the  short-term,  USDA  desires  “to  identify  potential  re¬ 
porting  systems  that  could  obtain  for  FNS  the  data  necessary  to  measure  the  nutrient 
content  of  school  lunches  against  the  nutrient  standards  in  Objective  2.1  of  FNS’s 
strategic  plan.’’^^  Specifically,  FNS  staff  “need  to  aggregate  nutrient  analysis  data 

Table  1.2 

Menu  Planning  Systems  in  the  NSLP 


Menu  Planning  System 

NuMenus 
Assisted  NuMenus 

Traditional  food -based 
menu  planning 
Enhanced  food-based 
menu  planning 

Any  reasonable  approach 


_ Description _ 

Nutrient  analysis  done  by  SFA 
Nutrient  analysis  done  by  third  party  (e.g.,  State 
Agency  or  outside  contractor) 

Similar  to  old  system;  no  nutrient  analysis  except 
during  SMI  review 

Similar  to  old  system  but  with  more  fruits,  veg¬ 
etables,  bread,  and  grains;  no  nutrient  analysis  ex¬ 
cept  during  SMI  review 
Any  other  system  approved  by  USDA 


^  Statement  of  Work,  Child  Nutrition  Analysis  Projects,  Task  Order  #4. 
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from  all  SMI  reviews  in  a  state  so  that  the  aggregated  data  represents  the  nutrient 
content  of  all  reviewed  meals  in  the  state  [and]  aggregate  nutrient  analysis  data  from 
all  SMI  reviews  nationwide  so  that  the  aggregated  data  represents  the  nutrient  con¬ 
tent  of  all  reviewed  meals  nationwide.” 

OBJECTIVES  AND  APPROACH 

To  explore  issues  related  to  state  reporting  of  the  results  of  SFA-level  SMI  review, 
USDA  contracted  with  RAND  to  conduct  a  closed-response  telephone  survey  of  all  50 
states.  After  the  original  RAND  interview  protocol  was  developed  and  reviewed  by 
the  Education  Information  Advisory  Committee  (EIAC),  USDA  redesigned  the  task  to 
involve  in-person,  in-depth,  interviews  with  State  ChUd  Nutrition  Directors  in  seven 
volunteer  states.  This  change  was  made  to  respond  to  concerns  of  committee  mem¬ 
bers  and  other  State  Child  Nutrition  Directors  that  FNS  was  performing  the  survey  as 
a  precursor  to  mandating  a  nationwide  reporting  system  for  SMI  data.  Rather  than 
mandate  anything,  FNS  decided  to  look  more  carefully  at  how  some  states  are  per¬ 
forming  SMI  reviews  and  how  they  might  be  able  to  transmit  data  with  limited  addi¬ 
tional  burden. 

FNS  asked  seven  states  to  work  collaboratively  with  RAND — through  on-site,  in- 
depth  interviews — to  explore  possible  reporting  systems.  FNS  chose  the  seven  states 
to  be  used  for  the  case  studies.  Figure  1.1  shows  the  seven  states,  which  were  chosen 
to  represent  very  different  methods  for  performing  SMI  reviews  and  the  seven  FNS 
regions.  The  seven  volunteer  states  are  shaded  and  the  thick  black  lines  outline  the 
regions  they  represent. 


RANDMR1296-1  1 


Figure  1.1 — Seven  Volunteer  States  Selected  and  the  FNS  Regions 


^^Ibid.  Note  that  this  Statement  of  Work  refers  to  "reviewed  meals”  while  the  Strategic  Plan  refers  to 
school  meals  in  general.  See  Chapter  5  for  a  discussion  on  "meals  reviewed”  versus  “meals  served”  and 
statistical  sampling. 
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At  the  beginning  of  1999,  the  RAND  team  contacted  the  State  Child  Nutrition 
Directors  or  other  main  contact  in  each  of  these  seven  states  by  telephone  to 
schedule  the  interviews.  The  interview  protocols  for  the  site  visits  were  based  on  the 
interview  protocol  developed  for  the  original  telephone  survey,  with  additional 
probes  to  obtain  specific  details  on  how  the  SMI  review  process  worked  in  that  state 
and  the  likely  impact  of  a  data  request  from  FNS.  After  the  initial  telephone  contact, 
copies  of  the  interview  protocol  were  faxed  to  the  states.  State  contacts  were  asked  to 
complete  a  short  background  questionnaire  and  return  it  to  RAND  before  the 
scheduled  meeting.  The  short  background  questionnaire  included  questions  about 
the  number  of  SFAs  in  the  state  and  the  number  of  SMI  reviews  completed.  One  or 
two  RAND  staff  members  visited  each  of  the  states  for  an  in-depth  interview.  After 
the  interview,  the  RAND  team  compiled  the  state  official  answers  to  the  interview 
questions  and  sent  the  written  answers  back  to  the  states  for  comments,  giving  them 
the  opportunity  to  correct  any  misinformation  or  add  any  missing  information. 

INDIVIDUALS  SELECTED  FOR  INTERVIEWS  IN  THE  SEVEN  STATES 

In  each  state  except  California  and  Wisconsin,  the  State  Agency  Director  was  con¬ 
tacted  and  that  person  assisted  us  in  arranging  to  speak  to  other  key  people.  In 
California  and  Wisconsin,  the  original  contact  person  was  someone  within  the 
Agency,  though  not  the  director.  Again,  the  contact  person  arranged  a  meeting  at 
which  time  RAND  spoke  to  the  contact  and  to  other  important  individuals  in  the 
state.  See  Appendix  B  for  a  complete  list  of  the  individuals  interviewed  in  each  of  the 
seven  states. 

ORGANIZATION  OF  THIS  DOCUMENT 

This  document  describes  the  results  of  those  site  visits  and  their  implications. 
Chapter  2  discusses  what  data  elements  should  be  sent  to  FNS  to  make  analysis  pos¬ 
sible,  which  elements  could  be  useful  but  are  not  necessary,  and  which  elements  we 
recommend  against  collecting.  Chapter  3  describes  SMI  reviews  in  general  terms 
and  how  they  are  conducted  in  the  seven  volunteer  states.  Chapter  4  discusses  how 
the  data  elements  should  be  sent  to  FNS.  Chapter  5  discusses  some  broader  issues  in 
the  design  of  the  SMI  review  effort.  The  complete  interview  protocol  can  be  found  in 
Appendix  A  of  this  document.  The  completed  responses  to  the  interview  protocol  for 
ail  seven  states  can  be  found  m  an  appendix  volume. 


Chapter  Two 

WHAT  DATA  SHOULD  BE  PROVIDED? 


Although  the  states  are  required  to  conduct  SMI  reviews,  USDA-FNS  currently  re¬ 
ceives  no  information  on  the  results  of  the  reviews  or  even  which  reviews  have  been 
conducted.  The  main  purpose  of  this  study  was  to  explore  approaches  that  USDA 
could  use  to  collect  information  on  the  results  of  SFA-level  SMI  reviews,  while  impos¬ 
ing  minimal  burden  on  the  states  and  the  SFAs.  To  design  an  appropriate  system, 
USDA  needs  to  determine  what  information  should  be  reported  and  how  that  infor¬ 
mation  should  reach  USDA.  We  discuss  “what”  in  this  chapter  and  “how”  in  Chapter 
4. 

What  should  be  collected  depends  on  the  uses  USDA  plans  to  make  of  the  informa¬ 
tion  it  receives.  To  assess  progress  toward  the  items  in  its  strategic  plan,  USDA 
would  like  to  be  able  to  tabulate  the  share  of  meals  served  in  reviewed  schools  that 
meet  each  of  the  guidelines  (and  combinations  of  the  guidelines).  Furthermore, 
USDA  would  like  to  be  able  to  do  the  tabulations  by  state,  type  of  institution,  and 
age /grade  range. 

Among  the  items  we  discussed  with  State  Agency  staff  during  the  site  visits  was  a  list 
of  data  elements  to  be  collected  (see  the  complete  interview  protocol  in  Appendix  A). 
FNS  provided  the  initial  list.  During  the  discussion  with  the  states  and  during  our 
own  analysis,  we  reconsidered  the  list  of  elements  to  be  collected.  The  list  included 
in  Appendix  A  is  an  exhaustive  list  of  potential  data  elements  to  collect.  However,  it 
appears  that  not  aU  of  these  elements  are  necessary  and  some  of  them  probably 
should  not  be  collected  at  all. 

We  begin  by  describing  the  data  elements  that  we  recommend  that  USDA  should 
collect,  along  with  some  that  might  be  useful  but  that  are  not  necessary  for  analysis. 
We  then  describe  the  elements  that  were  considered  for  inclusion  (either  because 
they  were  suggested  by  USDA-FNS  or  by  a  state  during  an  interview),  but  that  we 
recommend  not  be  included. 

REQUIRED  ELEMENTS 

The  following  screen  graphically  depicts  the  list  of  data  elements  that  we  have  con¬ 
cluded  USDA  needs  to  perform  its  desired  tabulations. 
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What  Data  Should  Be  Provided?  1 1 


Type  of  SFA:  An  indicator  for  public  school  or  school  district,  private  school  or 
school  district,  residential  child  care  institution  (RCCI),  or  other  type  of  SFA  is  impor¬ 
tant  because  different  types  of  SFAs  may  be  better  prepared  for  SMI  reviews  than 
others.  All  the  states  to  which  we  spoke  indicated  that  RCCIs  are  not  ready  for  SMI 
because  of  lack  of  staff.  In  addition,  FNS  has  said  that  it  is  currently  focusing  its  re¬ 
sources  helping  SFAs  meet  SMI  requirements.  Therefore,  FNS  will  need  to  know 
whether  results  are  for  an  RCCI  or  not.  Although  not  necessarily  collected  as  part  of 
the  current  SMI  review,  this  data  element  should  be  very  easy  for  the  states  to  collect 
in  the  future.  The  software  should  include  a  menu  of  choices. 


Type  of SFA 

Answer  Code 

Public  School  or  District 

1 

Private  School  or  District 

2 

RCCI 

3 

Other 

9 

Menu  Planning  System  Used:  An  indicator  for  traditional  food-based  menu  plan¬ 
ning,  enhanced  food-based  menu  planning,  nutrient  standard  menu  planning,  as¬ 
sisted  nutrient  standard  menu  planning,  or  “any  reasonable  approach”  is  needed. 
Depending  on  the  type  of  menu  planning  system  used,  the  SFA  may  do  the  nutrient 
analysis  or  the  state  may  do  it.  Knowing  who  did  the  analysis  will  be  useful  if  FNS 
sees  consistent  problems  with  either  the  state’s  management  of  analysis  or  the  way 
schools  are  doing  analysis.  Also,  states  have  to  be  aware  of  the  menu  planning  sys¬ 
tem  used  in  order  to  proceed  with  the  SMI  review.  Therefore,  this  information 
should  be  readily  available  during  the  review  and  will  only  need  to  be  added  to  the 
information  recorded  as  part  of  the  review.  The  software  should  include  a  menu  of 
choices. 


Type  of  Menu  Plan 

Answer  Code 

Nutrient  Standard  Menu  Planning 

1 

Assisted  Nutrient  Standard  Menu  Planning 

2 

Enhanced  Food-Based 

3 

Traditional  Food-Based 

4 

Any  Reasonable  Approach 

9 

Analysis  Lunch  Only,  Breakfast  Only,  or  Lunch  and  Breakfast:  FNS  is  interested 
only  in  analyses  of  lunch  menus  for  this  project.  However,  some  reviews  and  analy¬ 
ses  are  completed  for  both  lunch  and  breakfast.  Therefore,  FNS  needs  to  know 
which  meals  are  covered  by  an  analysis.  This  information  will  be  provided  by  the 
SFA.  As  with  the  type  of  SFA  and  the  type  of  menu  plan,  it  will  merely  need  to  be 
added  to  the  information  recorded.  The  software  should  include  a  menu  of  choices. 


Meal  Analyzed 

Answer  Code 

Lunch  Only 

1 

Breakfast  Only 

2 

Lunch  and  Breakfast 

3 

Analysis  Weighted:  Some  analyses  are  weighted,  either  by  type  of  entree  served  or  by 
all  menu  items — in  this  case,  weighting  refers  to  allowing  for  differences  depending 
on  the  number  of  portions  of  a  particular  entree  or  other  item  that  are  served.  If 
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pizza  and  chicken  are  both  served,  but  pizza  is  served  to  75  kids  and  chicken  to  only 
25,  then  the  pizza  would  be  weighted  75  percent  and  the  chicken  only  25  percent  in 
the  analysis.  Knowing  whether  an  analysis  was  weighted  or  not  allows  USDA  to  make 
comparisons  among  similar  strategies.  For  SFAs  using  NuMenus  or  Assisted 
NuMenus  in  states  where  weighting  is  not  mandated,  the  state  will  have  to  find  out 
from  the  SFA  whether  weighting  was  used  or  not.  For  the  SFAs  using  other  menu 
planning  systems,  the  state  will  do  the  analysis  itself  and  will  know  whether  weight¬ 
ing  was  used.  The  software  should  include  a  menu  of  choices. 


Analysis  Weighted 

Answer  Code 

Yes,  entree  only 

1 

Yes,  all  menu  items 

2 

No 

3 

Review  Done  by  Grade  or  Age:  An  indicator  of  grade  or  age  review  is  needed  to  in¬ 
terpret  the  next  two  elements.  This  information  is  needed  to  apply  the  appropriate 
nutrient  standards  and  so  will  be  readily  available.  The  software  should  include  a 
menu  of  choices. 


Review  Type 

Answer  Code 

Age  Range 

1 

Grade  Range 

2 

Lowest  Grade/Age:  Lowest  age  or  grade  range  for  the  analysis  being  reported  is  im¬ 
portant  to  assess  which  standards  should  be  used  and  to  allow  for  comparisons  by 
age  or  grade  range.  Some  states  are  using  standardized  age/grade  ranges  and  others 
are  using  customized  ones.  This  will  allow  USDA  to  see  which  is  being  used.  This 


Lowest  Grade/Age 

Answer  Code 

P 

19 

K 

20 

1 

1 

2 

2 

3 

3 

4 

4 

5 

5 

6 

6 

7 

7 

8 

8 

9 

9 

10 

10 

11 

11 

12 

12 

13 

13 

14 

14 

15 

15 

16 

16 

17 

17 

18 

18 
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information  should  already  be  available  since  the  computer  software  requires  it  to 
determine  the  standards  against  which  the  SFA  data  should  be  compared.  A  menu  of 
answer  choices  should  be  included. 

Highest  Grade/Age:  Highest  age  or  grade  range  for  the  analysis  being  reported  is  im¬ 
portant  in  order  to  assess  which  standards  should  be  used  and  to  allow  for  compar¬ 
isons  by  age  or  grade  range.  This  information  should  already  be  available  since  the 
computer  software  requires  it  to  determine  the  standards  against  which  the  SFA  data 
should  be  compared.  The  menu  of  answer  choices  would  be  identical  to  the  one  for 
lowest  age/grade  range. 

Estimated  Average  Daily  Participation:  This  number  is  the  estimated  average  daily 
school  lunch  participation  during  the  SMI  review  week  in  the  school  that  adminis¬ 
tered  the  SMI  review  for  the  student  age/grade  range  covered  by  this  analysis.  This 
figure  can  be  used  as  a  weight  so  that  nutrient  analysis  results  from  different  schools 
or  different  states  can  be  aggregated.  With  this  weighting  element,  FNS  will  be  able 
to  calculate  the  nutritional  content  of  the  meals  that  the  average  child  in  the  re¬ 
viewed  schools  is  getting,  i.e.,  larger  schools  get  larger  weights.^ 

This  number  may  not  be  readily  available.  Though  SFAs  have  aggregate  data  on 
meals  served,  they  may  have  to  estimate  the  number  of  meals  served  for  a  given  age 
or  grade  range  represented  by  the  analysis.  This  information  can  be  derived  from  the 
same  sources  used  to  report  the  number  of  meals  served  in  the  Coordinated  Review 
Effort.  Given  the  type  of  analysis  that  FNS  will  be  able  to  do  with  these  data,  an  exact 
number  is  probably  not  critical.  Instead,  an  estimate  by  the  SFA  seems  sufficient. 
This  is  the  one  required  data  element  that  may  cause  a  burden  to  some  of  the  states. 

Actual  Output  from  Analysis  Software:  These  numbers  are  the  data  on  the  eleven 
items  that  the  USDA  requires — calories,  fat,  saturated  fat,  protein,  calcium,  iron,  vi¬ 
tamin  A,  vitamin  C,  sodium,  cholesterol,  and  fiber.^  The  software  provides  quanti¬ 
tative  measures  of  all  eleven  elements.  This  information  would  come  directly  from 
the  output  of  the  nutrient  analysis. 

OPTIONAL  DATA  ELEMENTS 

SFA  Identifier:  A  unique  identifier  for  each  SFA  is  needed  to  distinguish  the  records 
in  the  database.  The  SFA  agreement  number  is  the  natural  candidate,  but  states  are 
somewhat  anxious  that  identifying  an  SFA  in  any  way,  even  with  a  random  identifier, 
may  allow  FNS  to  find  and  punish  SFAs  with  poor  results  on  nutrient  analyses.  To 
eliminate  this  possibility,  the  FNS  software  could  automatically  assign  a  numeric  ID 
number  (e.g.,  the  sequence  number).  If  states  wanted  to  provide  this,  they  could  use 


^  See  the  discussion  in  Chapter  5  about  whether  we  want  to  represent  meals  reviewed  or  meals  served. 

^  WhOe  FNS  does  not  have  quantitative  measures  for  sodium,  cholesterol,  and  fiber,  the  regulations  say 
that  the  states  should  show  a  decrease  in  sodium  and  cholesterol  and  an  increase  in  fiber.  This  regulation 
can  be  met  by  creating  quantitative  measures  for  these  items  within  the  state  or  by  showing  changes  from 
one  SMI  cycle  to  the  next. 
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the  SFA  identifier  that  they  already  have  in  place  with  a  year  appended  in  order  to 
make  it  unique  for  each  analysis  or  they  could  create  an  ID  number  in  numeric  order 
as  the  reviews  are  completed.  If  a  state  decides  to  include  this,  the  burden  would  be 
minimal. 

Review  Week:  The  dates  for  the  week  in  which  menus  were  reviewed  for  the  nutrient 
analysis  may  be  useful  to  see  any  seasonality  effect  on  the  analysis.  For  example,  if 
an  analysis  is  done  the  week  of  Thanksgiving,  the  school  may  serve  a  special  holiday 
meal,  which  is  high  in  fat.  This  data  element  should  not  be  difficult  to  get  because  it 
should  be  in  the  NutriKids^  output.  However,  because  of  peculiarities  of  NutriKids,  if 
too  many  reviews  happen  on  the  same  week,  the  reviewers  may  have  to  change  the 
dates  when  entering  them  since  NutriKids  can  handle  only  a  limited  number  of  re¬ 
views  in  a  given  week.  Therefore,  even  dates  of  the  review  week  may  be  somewhat 
difficult. 

Contact  Person:  The  name  of  someone  FNS  can  call  with  any  questions.  This  con¬ 
tact  could  be  the  same  person  for  all  analyses  from  each  state  or  could  be  different 
people  based  on  who  did  the  analysis.  In  either  case,  this  should  be  relatively  easy 
for  states  to  provide,  should  they  decide  to  do  so.  The  states  suggested  that  the  con¬ 
tact  person  be  the  State  Director.  This  data  element  could  be  optional  because  FNS 
indicates  that  it  does  not  currently  have  the  staff  for  going  back  to  respondents  to 
clarify  information.  If  there  were  questions  that  absolutely  had  to  be  answered,  FNS 
would  go  back  to  the  State  Director  for  clarification.  The  State  Directors  also  prefer 
this  approach. 

Clarification  of  “Any  Reasonable  Approach”:  A  brief  description  of  any  nonstandard 
menu  planning  system  used  could  be  optional  because  few  SFAs  use  "any  reasonable 
approach”  as  a  menu  planning  system.  In  addition,  across  SFAs  that  use  “any  rea¬ 
sonable  approach,”  there  is  little  consistency.  However,  we  include  it  as  an  optional 
data  element  because  it  may  be  useful  to  know  the  type  of  system  used,  especially  if 
the  results  from  the  analysis  seem  different  from  other  analyses  in  the  same  state  or 
in  schools  of  similar  size  and  type.  Even  if  the  descriptions  do  not  have  any  consis¬ 
tency  across  SFAs,  they  could  be  useful  on  an  individual  basis.  Since  SFAs  have  de¬ 
veloped  their  own  systems  in  this  case,  giving  clarification  on  the  approach  they  use 
should  be  very  straightforward. 

Urban  or  Rural  Area:  A  designation  for  urban  or  rural  area  is  probably  best  done 
with  U.S.  Census  Bureau  designations,  where  an  urban  area  comprises  “all  territory, 
population,  and  housing  units  in  urbanized  areas  and  in  places  of  2,500  or  more  per¬ 
sons  outside  urbanized  areas.  This  data  element  would  allow  FNS  to  differentiate 
between  large  schools  in  urban  and  rural  areas  since,  as  several  states  pointed  out, 
schools  in  rural  areas  may  be  less  well  prepared  for  SMI  reviews.  Most  SFAs  will 
probably  know  whether  the  area  they  are  located  in  is  designated  urban  or  rural.  If 


^  We  have  specifically  used  NutriKids  in  our  text  because  everyone  we  spoke  to  was  using  NutriKids.  Other 
software  packages  are  acceptable  according  to  FNS,  but  we  have  not  addressed  them  because  we  heard 
nothing  about  them  in  our  interviews. 

'*From  the  U.S.  Census  Bureau  web  page  at  bttp://www.census.gov/population/censusdata/urdef.txt. 
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not,  getting  this  information  from  the  U.S.  Census  Bureau  is  easy.  The  designation 
would  just  need  to  be  added  to  the  information  sent  to  the  state. 

Size  of  District:  A  measure  of  the  size  of  the  district — using  either  enrollment  or 
number  of  schools.  Enrollment  would  be  better  since  schools  can  vary  widely  in  size. 
However,  number  of  schools  could  be  useful  if  no  enrollment  figures  were  available. 
In  either  case,  the  SEA  would  have  to  provide  this  number.  Most  SFAs  will  probably 
know  enrollment  figures  and  should  be  able  to  provide  them  readily  to  the  state. 
This  data  element  would  be  useful  for  comparing  districts.  As  several  states  pointed 
out,  it  would  not  be  fair  to  compare  districts  with  100  kids  to  those  witli  100,000  kids. 

Comments:  Any  additional  comments  or  clarifications  that  the  state  wishes  to  make 
on  the  analysis  on  which  they  are  reporting  could  be  input  here.  Including  this  field 
gives  the  states  a  place  to  document  any  strange  circumstances  surrounding  a  par¬ 
ticular  review. 

DATA  ELEMENTS  THAT  SHOULD  NOT  BE  INCLUDED 

Several  other  data  elements  were  suggested  either  by  FNS  or  by  the  states.  We  rec¬ 
ommend  against  requiring  that  they  be  reported.  These  elements  include  the  name 
of  the  person  performing  the  analysis,  the  date  the  review  was  closed,  and  the  stan¬ 
dards  with  which  the  quantitative  elements  are  being  compared. 

Name  of  Reviewer:  The  states  were  very  strongly  opposed  to  providing  the  names  of 
reviewers,  consultants,  or  subcontractors.  The  State  Directors  consistently  stated 
that  since  it  was  their  responsibility  to  oversee  the  SMI  process,  FNS  should  contact 
them  with  any  questions  once  the  data  are  submitted.  Additionally,  if  FNS  examines 
the  data  annually,  some  of  the  reviews  would  be  a  year  or  more  old.  Thus,  even  if 
FNS  had  questions,  it  is  likely  that  the  states  would  no  longer  have  the  answers,  so  a 
contact  person  would  be  irrelevant. 

Dates  of  Review:  The  states  reported  that  providing  an  ending  date  for  a  review 
could  be  very  difficult,  though  they  could  probably  provide  a  start.  It  is  often  hard  to 
determine  when  the  review  is  complete.  A  review  may  be  complete  when  a  correc¬ 
tion  plan  is  implemented,  when  a  new  analysis  is  run,  when  a  closeout  letter  is  sent, 
depending  on  the  state. 

Standards:  USDA  suggested  requiring  the  states  to  provide  the  standards  against 
which  the  data  from  nutrient  analysis  are  compared.  The  states,  however,  noted  that 
in  most  cases  USDA  mandates  the  standards.  Thus,  knowledge  of  the  type  of  menu 
planning  system  used  and  the  age  or  grade  range  is  usually  enough  to  identify  the 
applicable  standard.  The  exception  is  states  that  are  using  a  menu  planning  system 
under  “any  reasonable  approach”  where  FNS  will  not  have  the  standards.  In  this 
case,  it  would  be  sufficient  for  the  state  to  submit  the  standards  to  USDA  annually 
along  with,  but  separate  from,  the  nutrient  analysis  results.  It  seems  unnecessary 
and  potentially  error  prone  to  require  the  states  to  resubmit  the  standards  with  each 
analysis.  One  possibility  would  be  for  USDA’s  program  to  pre-fill  the  standards  into 
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the  worksheet.  Since  NutriKids  can  automatically  assign  the  standards  given  the 
age/grade  range,  it  should  not  be  too  difficult  to  allow  this  to  happen  in  other  soft¬ 
ware. 


Chapter  Three 

CURRENT  REVIEW  PROCEDURES 


USDA  desires  to  understand  each  state’s  ability  to  report  the  required  information 
and  to  develop  a  system  that  would  impose  the  minimal  burden  on  the  states.  To  do 
so,  it  is  crucial  that  USDA  understand  the  systems  that  states  are  currently  using  to 
perform  their  SMI  reviews  and,  in  particular,  the  flow  of  the  required  information 
from  the  states  to  USDA  from  the  time  the  nutritional  analysis  is  completed.  This 
chapter  includes  this  information  for  the  seven  volunteer  states.  The  chapter  first 
describes  the  general  SMI  review  process  and  then  summarizes  the  process  in  each 
of  the  seven  states. 


OVERVIEW  OF  SMI  REVIEW  PROCESS  IN  THE  SEVEN  STATES 

When  we  interviewed  individuals  within  the  states  about  the  process  they  use  to 
conduct  SMI  reviews,  we  discovered  some  similarities.  Figure  3.1  shows  the  basic 
steps  for  completing  an  SMI  review  across  the  seven  states.  The  solid  arrows  indicate 
steps  that  every  state  follows.  The  dashed  hnes  indicate  steps  that  may  occur  in  some 
states  or  for  some  analyses,  but  that  are  not  always  present.  This  figure  does  not  in¬ 
clude  every  step  that  every  state  takes  in  completing  SMI  reviews.  Instead,  it  gives  a 
basic  outline  of  the  steps  that  most  states  use. 

In  every  state,  the  SFA  provides  specific  background  information  for  the  review  week. 
If  the  SFA  is  not  using  a  nutrient-based  menu  planning  system,  this  information  in¬ 
cludes  menus,  recipes,  and  production  records  to  the  state  or  a  contractor.  For  SFAs 
using  nutrient-based  plans,  they  also  provide  the  nutrient  analysis  for  review.  In 
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Figure  3.1 — Basic  Steps  in  Completing  the  SMI  Review  Across  States 
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some  cases,  this  information  is  sent  prior  to  the  SMI  site  visit,  and  in  other  cases  it  is 
collected  during  the  site  visit;  but  the  information  is  always  required  from  the  SFA. 
Sometimes  the  SFA  does  not  have  everything  available  prior  to  the  visit,  and  it  re¬ 
quires  several  weeks  or  even  months  after  the  visit  to  collect  all  the  necessary  infor¬ 
mation.  Once  the  background  information  is  collected,  it  is  sent  to  the  person  re¬ 
sponsible  for  completing  the  nutrient  analysis.  In  some  cases,  the  same  person  who 
collected  the  information  performs  the  nutrient  analysis.  In  other  cases,  the  infor¬ 
mation  is  sent  to  another  person — either  within  the  state  agency  or  at  an  outside 
contractor — for  the  analysis.  When  the  analysis  is  complete,  a  correction  plan  is  de¬ 
veloped  with  the  SFA,  and  the  review  is  closed. 

If  an  SFA  is  using  a  nutrient-based  menu  planning  system,  often  the  state  authority 
merely  collects  the  output  of  the  analysis  program.  The  analysis  is  usually  not  re¬ 
done  by  the  state.  Instead,  the  reviewer  examines  the  completed  analysis  sent  by  the 
SFA. 

HOW  THE  PROCESS  WORKS  WITHIN  EACH  OF  THE  SEVEN  STATES 

Each  state  operates  slighdy  differently,  and,  even  within  a  given  state,  reviewers  may 
alter  this  process  somewhat.  To  understand  that  variation  across  states,  we  present 
case  studies  for  seven  states  based  on  our  site  visits.  These  case  studies  consider  the 
following  issues:  who  collects  background  information,  who  performs  analyses, 
when  analyses  are  sent  to  the  state,  how  and  where  information  on  analyses  is  kept, 
and  how  difficult  it  would  be  to  send  information  to  FNS. 

California 

In  California,  reviewers  collect  hard  copies  of  the  menus,  recipes,  and  other  informa¬ 
tion  from  the  SFAs  during  the  site  visit.  The  reviewers  are  located  throughout  the 
state,  and  all  work  for  the  California  Department  of  Education.  In  most  cases,  the 
reviewers  send  the  information  to  the  Department  of  Education,  Nutrition  Standards 
Unit,  where  data  entry  staff  enter  it  into  the  NutriKids  program,  but  reviewers  will 
occasionally  do  the  data  entry  and  perform  the  nutrient  analyses  themselves.  Even 
when  a  reviewer  does  the  analysis,  it  is  reviewed  again  by  the  Nutrition  Standards 
Unit.  If  the  data  entry  staff  find  any  information  missing,  they  may  consult  with  the 
reviewers  or  foUow  up  with  the  SFA  themselves. 

Analyses  that  are  completed  at  the  state  office  are  not  revised.  Analyses  done  by  the 
reviewers  may  be  revised  if  the  Nutrition  Standards  Unit  personnel  find  problems, 
but  this  happens  only  rarely.  If  an  SFA  uses  NuMenus  or  Assisted  NuMenus,  the  re¬ 
viewers  will  examine  the  analysis,  but  they  wifi  not  redo  it.  Once  the  analyses  are 
finished,  they  are  kept  at  the  state  office  both  in  hard  copy  and  in  electronic  format. 
California  has  started  logging  the  results  of  the  nutrient  analyses  into  a  Microsoft 
Access  database  with  the  1998-1999  reviews.  Given  that  the  state  already  has  a 
database  set  up  and  has  agreed  to  revise  the  database  to  include  whichever  data  ele¬ 
ments  FNS  requires,  it  would  be  a  relatively  easy  task  for  California  to  report  nutrient 
analysis  data. 


Current  Review  Procedures  19 


Once  an  analysis  is  completed,  an  improvement  plan  is  developed  in  conjunction 
with  the  SFA.  Even  if  the  SEA  meets  all  the  nutritional  requirements,  there  is  usually 
some  sort  of  improvement  plan.  When  the  improvement  plan  is  accepted  by  the 
SFA,  the  review  is  considered  closed. 

There  are  1,496  SFAs  in  the  state  of  California.  As  of  June  1999,  206  (14  percent)  SMI 
reviews  had  been  completed  and  recorded  at  the  state  office.  However,  the  actual 
number  of  reviews  completed  in  June  1999  was  307  (21  percent),  but  some  reviews 
had  not  yet  been  recorded  when  we  talked  to  the  California  staff.  The  SFAs  in 
California  are  using  a  variety  of  menu  planning  systems.  About  20  percent  of  SFAs 
use  nutrient-based  plans,  more  than  35  percent  use  enhanced  food-based  plans,  20 
percent  use  traditional  food-based  plans,  and  25  percent  use  “any  reasonable  ap¬ 
proach.”  Although  USDA  no  longer  requires  that  analyses  be  weighted,  some 
California  analyses  continue  to  be  weighted. 

Georgia 

In  Georgia,  state  employees,  called  consultants,  perform  the  SMI  reviews  with  the 
help  of  secretaries.  (The  title  consultant  is  used  in  Georgia,  Nebraska,  and  Wisconsin 
for  reviewers  who  work  for  the  State  Agency).  These  employees  work  for  the  Georgia 
Department  of  Education,  but  they  have  offices  throughout  the  state  to  ease  the 
performance  of  the  site  visits.  In  the  future,  the  state  would  like  to  hire  a  subcontrac¬ 
tor  to  help  with  nutrient  analysis.  According  to  the  state  staff,  there  is  not  currently 
budget  to  do  so.  The  consultants  and  secretaries  are  responsible  for  collecting  aU  raw 
information  from  the  SFAs  during  the  site  visits.  The  secretaries  enter  the  data  into 
NutriKids,  and  the  consultants  do  the  analyses. 

Once  the  analysis  is  completed,  an  improvement  plan  is  formulated  and  the  SFA 
changes  its  menus  accordingly.  The  analysis  is  then  redone  based  on  these  changes. 
About  50  percent  of  SFAs  require  a  second  analysis  and  second  site  visit.  Once  the 
improvement  plan  is  implemented  and  the  analysis  is  redone,  the  review  is  com¬ 
plete. 

Georgia  does  not  currendy  have  any  state  database  that  records  the  results  of  the  nu¬ 
trient  analyses,  and  funds  are  not  available  to  create  such  a  database.  Hard  copies  of 
the  analyses  are  kept  at  the  state  offices  and  are  not  maintained  in  the  field  offices. 
No  soft  (computer  readable)  copies  of  analyses  are  currently  kept  on  file.  Currendy, 
it  would  be  a  fair  amount  of  work  for  Georgia  to  devise  a  method  for  repordng  data  to 
FNS.  They  do  not  have  any  system  in  place  to  enter  the  data  from  the  nutrient  analy¬ 
ses  into  a  database  of  any  sort.  Everything  would  have  to  be  keypunched  from  the 
hard  copies  unless  there  was  a  way  to  save  the  information  direcdy  from  NutriKids. 

The  Georgia  Department  of  Education  is  responsible  only  for  SMI  reviews  in  public 
SFAs,  so  it  must  review  180  SFAs.  As  of  July  1999,  when  we  talked  to  the  state  staff, 
they  had  completed  reviews  on  78  SFAs  (43  percent).  SFAs  in  Georgia  are  almost  all 
(95  percent)  using  traditional  food-based  menu  plans.  Georgia  requires  analyses  to 
be  weighted  by  type  of  entree  served. 
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Nebraska 

Unlike  t±ie  other  six  states,  Nebraska  performs  the  nutrient  analysis  prior  to  the  site 
visit  of  the  SFA.  Several  months  before  the  visit,  a  consultant  who  works  for  the  State 
Agency  calls  the  SFA  and  requests  all  the  necessary  information.  If  the  SFA  uses  a 
food-based  system,  it  sends  the  menus,  recipes,  etc.,  to  an  independent  contractor  to 
do  the  analysis  and  the  contractor  forwards  the  completed  review  to  the  consultant. 
If  the  SFA  uses  a  nutrient-based  system,  it  sends  the  analysis  itself  to  the  consultant 
for  review. 

The  consultant  then  brings  the  completed  nutrient  analysis  to  the  site  visit.  Since  the 
nutrient  analysis  is  already  complete,  at  the  visit  the  consultant  can  then  give  com¬ 
ments  to  the  SFA  and  ask  for  an  improvement  plan. 

Once  the  state  reviews  the  information  from  the  analysis,  either  by  the  contractor  or 
directly  from  the  SFA,  it  is  logged  into  a  Microsoft  Access  database.  Hard  copies  of 
the  analyses  are  also  kept  at  the  state  office.  The  electronic  copies  of  the  analyses  are 
not  kept  past  the  end  of  the  school  year  in  which  they  were  done.  Since  the  state  al¬ 
ready  has  a  Microsoft  Access  database  that  could  easily  be  modified  to  fit  the  needs 
of  FNS,  Nebraska  should  not  have  a  problem  supplying  the  necessary  data  in  the  fu¬ 
ture,  assuming  it  can  be  done  directly  from  the  database. 

Nebraska  has  506  SFAs  in  the  state  and  had  completed  reviews  on  103  (20  percent)  as 
of  July  1999.  SFAs  in  Nebraska  are  mostly  using  food-based  menu  planning  systems, 
with  about  80  percent  using  enhanced  food-based  and  16  percent  using  traditional 
food-based.  The  rest  are  using  nutrient-based  systems.  Nebraska  requires  analyses 
to  be  weighted  by  type  of  entree. 

New  Jersey 

In  New  Jersey,  field  staff,  who  work  for  the  New  Jersey  Department  of  Agriculture,  do 
all  site  visits.  They  hand -carry  information  from  the  SFA  back  to  the  state  office  and 
then  send  it  to  an  outside  contractor  (located  in  Ohio)  for  the  nutrient  analysis.  The 
contractor  sends  the  completed  analyses  back  to  the  state  office,  and  the  reviewer 
has  an  exit  conference  with  the  SFA  to  discuss  improvement  plans.  The  reviewer 
then  sends  a  letter  to  the  district  outlining  an  improvement  plan.  Finally,  once  the 
improvement  plan  is  implemented,  the  SFA  receives  a  closeout  letter  signaling  that 
the  review  is  complete. 

New  Jersey  does  not  currently  have  any  method  for  aggregating  the  nutrient  analy¬ 
ses.  State  officials  stated  that  they  preferred  to  wait  and  see  what  information  FNS 
requires  them  to  submit  before  developing  a  system  to  use  for  aggregation.  They 
would  like  FNS  to  provide  a  protocol  for  a  database  before  they  create  anything.  All 
hard  copies  of  analyses  are  kept  at  the  state  office.  Currently,  no  electronic  informa¬ 
tion  is  kept  on  reviews,  so  New  Jersey  would  also  need  to  devise  a  system  to  send  the 
data  to  FNS  electronically.  State  staff  are  willing  to  do  what  FNS  needs  but  are  wait¬ 
ing  for  guidance  from  FNS. 
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New  Jersey  has  716  SFAs  and  had  completed  reviews  on  100  (14  percent)  as  of  August 
1999.  About  90  percent  of  SFAs  in  New  Jersey  are  using  enhanced  food-based  menu 
planning  systems.  The  other  10  percent  are  using  either  traditional  food-based  or 
nutrient-based  systems.  Although  no  longer  required  by  federal  regulations,  the 
state  of  New  Jersey  requires  nutrient  analyses  to  be  weighted  by  type  of  entree 
served. 

New  York 

In  New  York,  individual  reviewers  who  work  for  the  New  York  Department  of 
Education  collect  the  necessary  information  from  the  SFAs  during  the  site  visits  using 
a  statewide  protocol.  The  level  of  technical  assistance  varies,  however,  based  on  the 
review  and  the  SFA’s  need.  If  an  SFA  is  using  a  food-based  system,  the  reviewer  per¬ 
forms  the  nutrient  analysis.  If  the  SFA  is  using  a  nutrient-based  system,  the  SFA  is  re¬ 
sponsible  for  its  own  nutrient  analysis  and  the  reviewer  will  assess  that  analysis  as 
well  as  the  SFA’s  competence  in  doing  the  analysis.  After  the  analysis  is  reviewed,  a 
correction  plan  is  developed  with  the  SFA  and  a  hard  copy  of  the  final  review  is  filed 
at  the  State  Agency. 

New  York  is  in  the  process  of  creating  a  Microsoft  Access  database  to  aggregate  the 
nutrient  analysis  information.  However,  making  the  database  operational  is  not  a 
priority.  State  officials  said  that  they  prefer  to  wait  until  FNS  issues  its  requirements. 
In  the  meantime,  the  analyses  are  kept  in  hard  copy  at  the  state  office  and  a  summary 
of  each  analysis  is  kept  in  electronic  format.  However,  this  summary  does  not  in¬ 
clude  all  the  information  that  FNS  will  likely  require  on  SMI  reviews.  Since  New  York 
has  already  started  creating  a  database,  it  should  not  be  a  difficult  task  to  modify  it  to 
contain  all  the  necessary  data  elements  and  then  export  the  data  for  FNS. 

New  York  has  1,275  SFAs  and  had  completed  between  93  (7  percent)  and  113  (9  per¬ 
cent)  SMI  reviews  as  of  August  1999.  About  50  percent  of  SFAs  in  New  York  are  using 
nutrient-based  analyses,  in  part  because  the  state  staff  "put  a  lot  of  effort”  into 
training  school  personnel  on  how  to  perform  nutrient  analyses  correctly.  The  other 
50  percent  are  fairly  evenly  split  between  enhanced  and  traditional  food-based  sys¬ 
tems.  ^  New  York  does  require  analyses  to  be  weighted. 

Texas 

Texas  has  a  different  system  for  completing  SMI  reviews  than  the  other  six  states. 
Unlike  the  other  state  agencies,  the  Texas  Education  Agency  (TEA)  is  not  heavily  in¬ 
volved  in  SMI  reviews.  Instead,  TEA  has  divided  Texas  among  20  Educational  Service 
Centers  (ESCs),  which  are  responsible  for  SMI  reviews  as  well  as  technical  assistance 
for  disabled  students,  multicultural  studies,  and  the  Public  Education  Information 


^These  were  the  numbers  that  we  were  given  at  our  site  visit  in  August  1999.  New  York  has  since  revised 
these  numbers:  128  reviews  (10  percent)  completed  as  of  August  1999.  About  40  percent  of  their  SFAs  are 
using  nutrient-based  analyses.  The  enhanced  food-based  option  was  selected  by  37  percent  of  the  SFAs, 
and  the  traditional  plan  was  selected  by  23  percent  of  the  SFAs.  The  numbers  in  Table  3.1  also  reflect  what 
we  heard  in  August  1999  and  would  be  updated  by  this  information. 
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Management  Systems,  which  collects  student  demographic  and  academic  perfor¬ 
mance,  personnel,  financial,  and  organizational  information.  Each  ESC  has  a  Child 
Nutrition  Office  that  is  responsible  for  the  SMI  reviews.  Other  personnel  within  the 
ESC  perform  the  other  services  mentioned.  The  ESCs  are  independent  entities  that 
perform  reviews  and  provide  technical  assistance  to  SFAs.  Therefore,  it  is  difficult  to 
construct  a  comprehensive  description  of  how  the  SMI  review  process  works  in  Texas 
because  each  ESC  is  autonomous  and  has  its  own  method  for  performing  reviews. 
TEA  does  not  currently  receive  any  nutrient  analysis  information  from  the  ESCs,  and 
therefore  no  aggregate  information  exists  at  the  state  level. 

The  ESC  staff  do  not  work  for  the  state  but  are  employees  of  the  ESC,  which  is  an  in¬ 
dependent  entity.  We  spoke  to  the  ESC  staff  in  Region  XIII,  which  includes  the  area 
around  Austin.  In  that  region,  the  ESC  staff  perform  all  site  visits  and  the  nutrient 
analysis.  The  standard  process  is  for  the  ESC  staff  to  send  an  initial  letter  to  the  SEA 
listing  the  information  it  will  need  for  the  review.  Then,  a  second  letter  is  sent  in¬ 
forming  the  SEA  of  the  dates  for  the  review  week  and  the  date  of  the  site  visit.  Even  in 
cases  where  an  SEA  is  using  a  nutrient-based  system,  ESC  staff  will  redo  the  nutrient 
analysis  rather  than  rely  on  the  SFA’s  analysis,  thus  doing  nutrient  analyses  on  every 
SEA  that  they  visit. 

Because  most  districts  are  not  ready  to  complete  a  final  analysis  of  the  nutrients  in 
their  meals,  the  analyses  often  have  to  be  done  several  times  before  they  are  correct. 
Therefore,  officials  in  Texas  referred  to  the  analysis  as  a  “living  document.”  During 
the  1998-1999  school  year,  almost  all  analyses  in  Region  XIII  were  revised  after  they 
were  initially  done.  ESC  staff  reported  that  this  was  because  it  was  the  first  year  they 
did  SMI  reviews  and  SFAs  were  still  getting  used  to  the  process. 

A  hard  copy  of  the  analysis  for  each  SEA  is  kept  by  the  ESC.  The  SFAs  themselves  are 
also  required  to  keep  a  copy  of  the  analysis  and  the  improvement  plan.  General  in¬ 
formation  about  all  analyses  performed  is  sent  to  the  state  annually.  However,  no 
SFA-level  information  is  sent  to  the  state.  Of  all  the  states  we  spoke  to,  Texas  will 
probably  have  the  most  difficult  time  supplying  FNS  with  nutrient  analysis  data. 
Since  the  state  staff  would  want  to  see  the  data  before  sending  it  on,  they  would  first 
need  to  collect  all  the  analyses  from  the  20  ESCs  at  TEA  and  then  input  the  data  for 
FNS  or  have  the  ESCs  input  the  data  and  forward  it  to  TEA.  In  either  case,  the  pro¬ 
cess  would  require  several  steps  that  TEA  is  not  currently  taking. 

The  TEA  and  the  ESCs  in  Texas  are  responsible  only  for  public  SFAs.  The  20  ESCs 
must  do  SMI  reviews  on  1,100  public  SFAs.  Because  no  information  on  individual 
reviews  is  sent  to  the  state,  it  is  not  possible  to  say  how  many  reviews  have  been  done 
to  date  without  talking  to  each  ESC  individually  (which  we  did  not  do).  About  50  per¬ 
cent  of  SFAs  in  Texas  are  using  traditional  food-based  menu  planning  systems.  Of 
the  others,  about  30  percent  use  enhanced  food-based  and  about  20  percent  use 
nutrient-based  systems.  Texas  does  not  require  analyses  to  be  weighted. 
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Wisconsin 

In  Wisconsin,  a  consultant  collects  all  the  menus,  recipes,  and  other  supporting  doc¬ 
uments  from  the  SFA  during  the  site  visit,  unless  these  materials  were  sent  prior  to 
the  visit.  More  than  half  the  schools  send  the  packets  ahead  of  time.  As  in  Georgia 
and  Nebraska,  the  state  employees  are  called  consultants.  Either  the  consultant 
performing  the  site  visit  or  a  central  consultant  at  the  state  office  then  completes  the 
nutrient  analysis.  If  materials  are  sent  prior  to  the  visit,  then  the  analysis  can  be 
completed  in  time  to  discuss  it  at  the  site  visit.  Occasionally,  analyses  are  corrected  if 
the  school  is  using  a  nutrient-based  system  and  had  a  problem  doing  the  analysis  it¬ 
self  or  if  the  school  is  using  a  food-based  system  and  submitted  erroneous  informa¬ 
tion  for  analysis.  When  an  analysis  is  corrected  under  these  circumstances  the  up¬ 
dated  and  most  accurate  analysis  is  recorded  in  the  state  spreadsheet.  Once  the 
analysis  is  finished,  the  consultant  works  with  the  SFA  to  form  a  correction  plan. 
Once  the  consultant  has  determined  that  the  district  has  begun  the  correction  plan,  a 
closure  letter  is  sent  to  the  SFA. 

Wisconsin  is  currently  using  a  Microsoft  Excel  spreadsheet  to  enter  the  nutrient  anal¬ 
ysis  information  for  monitoring  and  analysis  at  the  state  level.  Hard  copies  of  the 
analyses  are  also  kept  at  the  state  office.  Assuming  FNS  would  accept  data  in  various 
formats,  it  should  be  a  simple  matter  for  Wisconsin  to  take  the  data  in  the  spread¬ 
sheet  and  modify  it  for  FNS. 

Wisconsin  has  900  SFAs  and  had  completed  reviews  on  312  (35  percent)  as  of  June 
1999.  Nearly  70  percent  of  SFAs  in  Wisconsin  use  enhanced  food-based  systems. 
Almost  30  percent  use  traditional  food-based  systems,  and  less  than  5  percent  use 
nutrient-based  systems.  Wisconsin  requires  that  analyses  be  weighted  by  aU  menu 
items. 


SUMMARY  OF  SMI  REVIEW  INFORMATION 

In  this  section  we  present  the  information  on  ail  seven  states  together.  Table  3.1  lists 
the  status  of  reviews  (June-August  1999)  in  terms  of  the  SFAs  in  each  state  as  of  the 
time  of  the  interview.  As  we  mentioned  before,  the  agencies  in  Georgia  and  Texas  are 

Table  3.1 

Summary  of  Status  of  SMI  Reviews  in  the  Seven  States 


State 

Number  of 
SFAs 

Number  of  SMI  Reviews 
Completed  as  of  Summer 
1999 

Percentage  of  SMI  Reviews 
Completed  as  of  Summer 
1999 

California 

1,496 

206  (307) 

14(21) 

Georgia 

180 

78 

43 

Nebraska 

506 

103 

20 

New  Jersey 

716 

100 

14 

New  York 

1,275 

93  (113) 

7(9) 

Texas 

1,100 

188 

17 

Wisconsin 

900 

312 

35 

NOTES:  Numbers  in  parentheses  indicate  the  estimated  number  or  percentage  of  re¬ 
views  completed  once  they  are  all  logged  in;  New  York  figures  are  the  revised  numbers 
sent  after  our  visit. 


24  State  Monitoring  of  NSLP  Nutritional  Content 


responsible  only  for  public  SFAs;  therefore,  the  number  of  SFAs  listed  for  those  states 
represents  only  public  SFAs.  The  agencies  in  the  other  states  are  responsible  for  both 
public  and  private  SFAs. 

Table  3.1  suggests  that  the  states  are  at  varying  points  in  their  SMI  review  efforts. 
Georgia  and  Wisconsin  have  completed  more  than  a  third  of  the  required  reviews. 
New  Jersey  and  New  York  have  completed  less  than  a  sixth. 

Interpretation  of  the  percent  of  reviews  completed  is  complicated  by  the  nature  of 
the  review  process.  Not  all  SMI  reviews  that  have  been  performed  had  been  logged 
into  state  systems  at  the  time  of  the  interviews.  Therefore,  some  states  had  to  esti¬ 
mate  the  number  of  reviews  that  were  done  in  the  1998-1999  school  year.  In  the 
table  below,  the  numbers  represent  the  actual  completed  reviews.  Numbers  in 
parentheses  represent  the  estimated  number  and  percent  of  reviews  that  will  be 
completed  once  all  reviews  for  that  school  year  are  logged  in  and  exit  conferences  are 
finished. 

Table  3.2  summarizes  the  information  collected  on  how  the  SMI  reviews  were  con¬ 
ducted  within  the  seven  states.  In  some  states  the  reviewer  does  the  nutrient  analy¬ 
sis.  In  other  states,  the  nutrient  analysis  is  done  by  someone  else — a  member  of  the 
central  office  or  an  outside  contractor.  Some  states  use  a  mixture  of  the  two  ap¬ 
proaches.  The  “Weighted  Analysis”  column  indicates  whether  a  state  is  using  that 
method. 

Most  of  the  states  received  the  data  from  each  nutrient  analysis  in  the  State  Agency 
office.  The  column  entitled  “State  Has  Data”  in  Table  3.2  shows  whether  the  state 
office  receives  the  data  and  whether  they  have  compiled  it  into  any  type  of  central 
database. 

All  State  Agency  personnel  or  State  Agency  contractors  in  these  seven  states  use 
NutriKids  software.  However,  SFAs  and  others,  such  as  food  service  management 
companies  that  perform  their  own  computer  analyses,  may  use  different  software. 
Therefore,  not  all  nutrient  analyses  in  each  state  are  necessarily  done  on  NutriKids. 

Table  3.2 

Summary  of  How  SMI  Reviews  Were  Performed  in  the  Seven  States 


State 

Who  Does  Analysis 

Reviewer  Other 

Weighted 

Analysis? 

State  Has 

Data?* 

Software  Used 

California 

X 

X 

Some 

Y,  DB 

NutriKids 

Georgia 

X 

X 

Y 

Y,  NDB 

NutriKids 

Nebraska 

X 

Y 

Y,  DB 

NutriKids 

New  Jersey 

X 

Y 

Y,  NDB 

NutriKids 

New  York 

X 

X 

Y 

Y,  DB 

NutriKids 

Texas 

X 

N 

N 

NutriKids 

Wisconsin 

X 

X 

Y 

Y,  DB 

NutriKids 

*DB  =  database  or  spreadsheet;  NDB  =  no  database  or  spreadsheet 
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Table  3.3  shows  the  reviewers’  employer  and  location.  In  every  state  except  Texas, 
the  reviewers  work  directly  for  the  State  Agency,  although  they  are  not  always  based 
at  the  central  office. 

In  this  table,  “State — Centralized”  refers  to  reviewers  who  work  out  of  the  main  office 
of  the  State  Agency.  “State — Distributed”  refers  to  reviewers  who  work  for  the  State 
Agency  but  are  located  in  other  parts  of  the  state  away  from  the  central  office. 
“Contract — Distributed”  refers  to  reviewers  who  do  not  work  for  the  State  Agency 
and  are  spread  out  around  the  state. 

Table  3.4  shows  the  summary  of  menu  planning  systems  that  districts  use  in  each  of 
the  seven  states.  In  five  of  the  seven  states,  more  than  80  percent  of  SFAs  are  using 
food-based  systems,  either  “Traditional”  or  “Enhanced.”  In  three  of  the  seven  states, 
more  than  15  percent  of  SFAs  are  using  “Nutrient  Analysis.”  In  New  York,  nearly  half 
of  the  SFAs  are  using  “Nutrient  Analysis.”  In  California  and  Texas,  nearly  one  in  five 
SFAs  are  using  “Nutrient  Analysis.” 


Table  3.3 

Summary  of  SMI  Reviewers’  Employer  and  Location  in  the  Seven  States 


State 

State — Centralized 

State — Distributed 

Contract — Distributed 

California 

X 

X 

Georgia 

X 

X 

Nebraska 

X 

New  Jersey 

X 

New  York 

X 

X 

X 

Texas 

X 

Wisconsin 

X 

Table  3.4 

Summary  of  Menu  Planning  Systems  Used  for  SMI  Reviews  in  the  Seven  States 


State 

Menu  Planning  System  Used 
(Rough  Estimates  of  Percent  of  Districts) 

Nutrient 

Analysis 

Enhanced 

Food-Based 

Traditional 

Food-Based 

Other 

California 

18 

37 

20 

25 

Georgia 

? 

? 

95 

0 

Nebraska 

4 

80 

16 

0 

New  Jersey 

3 

91 

6 

0 

New  York 

48 

28 

23 

0 

Texas 

17 

30 

53 

0 

Wisconsin 

3 

68 

28 

0 

NOTE;  ?  =  State  was  unable  to  provide  these  numbers. 
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Chapter  Four 

HOW  DATA  ELEMENTS  COULD  BE  COLLECTED 


Having  considered  what  should  be  reported  and  current  state  review  procedures,  in 
this  chapter  we  turn  to  the  core  of  this  report:  How  states  could  most  easily  report 
the  results  of  their  review  efforts  to  USDA.  We  begin  by  asking  two  questions:  Who 
should  do  the  reporting?  When  in  the  review  process  should  the  report  be  sent  to 
USDA?  We  then  consider  the  technical  details  of  how  the  reporting  should  be  done. 
On  the  state  side,  we  consider  file  formats  and  methods  for  forwarding  the  data  to 
USDA.  On  the  USDA  side,  we  consider  how  to  store  and  analyze  the  data. 
Throughout,  we  give  careful  consideration  to  minimizing  the  burden  on  the  states. 

WHO  SHOULD  REPORT  WHEN? 

Who  should  report  the  state's  review  information  and  when  in  the  review  process 
should  that  person  do  so?  To  consider  the  broadest  possible  range  of  approaches  to 
state  reporting,  we  want  to  begin  with  the  point  at  which  there  exists  a  nutrient  anal¬ 
ysis  that  could  be  provided  to  USDA.  To  do  that,  we  need  to  consider  when  a  review 
is  complete. 

As  discussed  in  the  previous  chapter,  state  procedures  vary.  For  the  purposes  of  this 
discussion,  it  is  enough  to  note  that  the  materials  from  the  review  are  collected  by 
someone  and  entered  into  the  nutrient  analysis  software  (sometimes  the  person 
making  the  on-site  visit,  sometimes  a  state  employee  in  a  central  office,  sometimes  a 
private  nutritionist).  The  results  of  the  nutrient  analysis  are  available  essentially  im¬ 
mediately  after  all  of  the  information  (i.e.,  menus  and  nutritional  content  of  ingredi¬ 
ents)  is  entered — a  few  seconds  to  do  the  analysis,  a  few  more  seconds  to  save  or 
print  the  results.  In  practice,  it  often  takes  several  cycles  of  data  collection  (by  field 
staff),  data  entry,  and  review  (by  a  senior  staff  person)  to  collect  all  of  the  necessary 
information  (menus,  specific  products  used,  nutritional  content  of  products)  and  to 
clarify  apparently  erroneous  information.  For  example,  the  data  entry  person  may 
begin  input  and  realize  that  a  menu  is  missing  or  a  food  item’s  nutrient  content  is  not 
included.  In  these  cases,  the  state  staff  contacts  the  SFA  again  for  this  information 
before  data  input  can  be  completed.  For  the  discussion  here,  that  is  an  unimportant 
detail. 

Similarly,  under  some  circumstances,  some  states  revise  their  reviews.  For  example, 
in  Georgia,  about  half  of  the  nutrient  analyses  are  redone  after  an  improvement  plan 
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is  implemented.  In  most  other  states,  analyses  are  revised  only  if  an  error  is  found, 
either  with  data  entry  or  with  the  materials  that  the  SFAs  provided. 

To  monitor  the  content  of  meals  actually  being  served,  USDA  would  want  the  results 
of  the  final  reviews.  Most  states  that  enter  the  nutrient  analysis  data  into  a  database 
or  spreadsheet  use  the  initial  review.  Thus,  it  should  be  possible  to  report  the  results 
of  the  review  immediately  after  completion  of  the  nutrient  analysis.  It  should  not  be 
necessary  to  wait  for  the  completion  of  any  revised  review,  nor  for  the  development 
of  the  improvement  plan,  nor  for  the  official  closing  of  the  review.  Since  only  the  ini¬ 
tial  analysis  is  considered  part  of  an  SMI  review  according  to  FNS,  states  should  only 
report  information  from  this  analysis  to  FNS.  Only  in  cases  where  an  error  was  found 
with  inputting  the  information  in  the  analysis  software  would  a  “revised”  analysis  be 
appropriate.  In  states  such  as  Georgia  where  analyses  are  revised  based  on  the  im¬ 
plementation  of  improvement  plans,  USDA  would  want  the  results  of  the  pre¬ 
revision  nutrient  analysis.  Therefore,  FNS  should  make  clear  in  directions  to  the 
states  that  the  original  analysis  is  the  appropriate  one  to  report,  even  if  this  is  not 
when  the  state  considers  the  review  completed. 

To  allow  for  the  broadest  range  of  options,  we  should  consider  having  any  of  the 
people  who  have  access  to  the  nutrient  analysis  after  its  completion  forward  the  re¬ 
view  to  USDA.  Depending  on  the  state,  the  list  of  such  people  might  include  the  per¬ 
son  who  inputs  the  data  and  performs  the  nutrient  analysis  (perhaps  the  field  staff 
person,  perhaps  a  state  employee  in  a  central  office,  perhaps  a  contractor),  the  per¬ 
son  who  reviews  the  computer  output,  or  the  clerk  who  files  the  analysis  (though  the 
clerk  will  often  not  get  the  file  until  the  review  is  complete,  sometimes  well  after  the 
nutrient  analysis  is  complete). 

In  practice,  the  consideration  of  such  options  was  uniformly  and  vehemently  re¬ 
jected  by  every  state  we  visited.  Senior  state  officials  all  said  that  they  were  respon¬ 
sible  for  the  SMI  reviews.  If  the  results  of  the  nutrient  analysis  were  to  be  reported  to 
USDA,  they  would  want  to  review  the  analysis  or  have  a  designated  person  in  their 
office  do  so  before  it  was  forwarded  to  USDA. 

This  state  reaction  has  important  and  unfortunate  consequences  for  USDA.  USDA’s 
stated  goal  is  to  have  states  report  data  with  minimal  additional  effort  or  disruption 
of  their  current  routine.  At  least  in  Texas,  these  two  goals  appear  to  be  mutually  ex¬ 
clusive.  The  Texas  Education  Agency  does  not  currendy  collect  the  results  of  reviews. 
Instead,  the  individual  reviews  are  held  by  the  Educational  Service  Centers  (ESCs). 
Nevertheless,  even  Texas  officials  stated  that  they  would  not  be  willing  to  forward  the 
results  of  nutrient  analyses  to  USDA  without  reviewing  them  centrally.  Thus,  any 
USDA  reporting  requirement  wiU  force  Texas  to  collect  and  review  the  nutrient  analy¬ 
ses  centrally. 

While  in  principle  we  could  consider  several  alternatives  as  to  when  in  the  process 
the  report  should  be  sent  to  USDA  and  by  whom,  in  practice,  it  seems  clear  that  re¬ 
porting  will  be  done  by  central  office  staff.  Furthermore,  this  line  of  reasoning 
strongly  suggests  that  the  reporting  will  be  done,  not  when  the  nutrient  analysis  is 
completed,  but  when  the  review  (including  the  completion  of  the  improvement  plan) 
is  completed.  Furthermore,  it  will  sometimes  be  true  that  by  the  time  the  review  is 
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completed,  there  will  be  not  only  the  initial  nutrient  analysis  but  also  a  revised  nutri¬ 
ent  analysis.  USDA  will  need  to  give  clear  instructions  as  to  which  review  it  wants. 
We  recommend  that  USDA  explicitly  request  the  initial  review. 

FILE  FORMATS  AND  STATE  DATABASES 

USDA’s  projected  uses  of  the  data  require  that  it  receive  not  aggregated  data  (as  is 
currently  true  for  Coordinated  Review  Efforts  (CREs)),  but  instead  information  on 
each  review.  Happily,  this  need  appears  to  be  consistent  with  state  desires.  States 
have  complained  that  they  have  little  use  for  the  aggregate  CRE  data  and  that  aggre¬ 
gating  the  results  is  therefore  an  unnecessary  burden. 

Having  tentatively  concluded  that  reviews  will  be  forwarded  to  USDA  by  some  mem¬ 
ber  of  the  state’s  central  office  staff,  we  turn  to  the  file  format  in  which  the  data 
should  be  forwarded  to  USDA  and  the  closely  related  issue  of  state  databases.  There 
appear  to  be  two  broad  approaches  to  these  issues: 

•  States  could  create  a  database  for  their  own  use.  At  some  interval  (presumably 
annually),  the  states  could  forward  the  content  of  the  database  to  USDA. 

•  Alternatively,  states  could  forward  the  information  to  USDA  review  by  review.  In 
this  case,  states  might  find  it  useful  to  access  USDA’s  database. 

We  consider  each  of  these  approaches  and  mixtures  of  the  two. 

State  Databases 

Some  states  (Cahfornia,  Nebraska,  New  York,  and  Wisconsin)  are  already  working  on 
their  ovm  local  databases  or  spreadsheets.  They  record  the  results  of  each  review 
into  the  database  as  it  is  completed.  They  then  use  the  database  locally  to  monitor 
the  status  of  reviews  and  the  results  of  the  reviews.  We  learned  of  databases  being 
constructed  in  Microsoft  Excel  (spreadsheet),  Microsoft  Access  (low-end  database), 
and  Oracle  (high-end  database).^ 

Several  other  states  indicated  that  they  expected  to  move  to  a  local  database  in  the 
intermediate  term.  One  state  (New  York)  already  had  a  local  database  project  un¬ 
derway.  Another  state  (New  Jersey)  was  deferring  its  local  database  project  until  it 
received  clearer  guidance  from  USDA  on  what  should  be  collected.  Some  states 
(New  York,  New  Jersey,  Georgia)  indicated  that  they  would  welcome  (and  perhaps 
expect)  USDA-provided  software  for  the  construction  of  a  local  database. ^ 

The  existence  of  a  local  database  is  useful  for  USDA  because  the  data  already  exists  in 
computerized  form  and  it  should  be  relatively  easy  for  the  state  to  forward  the  review 
information  to  USDA.  The  state  would  simply  forward  the  computer  file  to  USDA 


^Wisconsin  uses  an  Excel  spreadsheet;  California,  Nebraska,  and  New  York  are  in  various  phases  of  devel¬ 
oping  an  Access  database;  New  York  uses  Oracle  for  some  of  its  database  needs,  though  it  has  not  devel¬ 
oped  a  full  database  for  SMI. 

sample  local  database  is  available  at  http://wvw.rand.org/organization/drd/labor/foodnupolicy.html. 
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(perhaps  with  some  documentation  as  to  the  meaning  of  fields).  Some  simple  pro¬ 
gramming  might  be  needed  to  read  the  state  file  into  USDA’s  national  database,  but 
doing  so  would  be  relatively  straightforward.  Standard  (and  inexpensive)  software 
products  (e.g.,  DBMS/Copy,  Stat/Transfer^)  will  perform  the  file  format  translations 
(e.g.,  from  Excel  or  Access  into  Oracle  or  SAS).  Some  software  packages  will  do  the 
file  format  translations  internally  (e.g.,  SAS  will  read  Excel  and  SQL  databases  such  as 
Oracle). 

The  file  format  translations,  however,  are  only  the  start  of  the  task.  Once  the  files  are 
in  the  final  file  format,  there  is  then  the  issue  of  renaming  variables  and  recoding  val¬ 
ues  so  that  the  state  data  can  be  merged  into  the  USDA  database  to  yield  analyzable 
data.  For  this  task,  the  effort  required  by  USDA  to  read  the  data  into  its  database  will 
be  approximately  proportional  to  the  number  of  different  database  systems  the 
states  use.  As  long  as  the  file  formats  do  not  change  much  from  year  to  year,  50  states 
is  a  feasible  number  of  formats  to  process.  A  programmer  of  moderate  skill  could 
probably  do  most  formats  in  an  hour  or  two. 

Clearly,  however,  it  would  be  much  simpler  for  USDA  to  provide  the  states  with  a 
program  that  they  may  choose  to  use.  Several  states  said  they  preferred  that  USDA 
provide  the  program,  and  if  it  did  so  they  would  use  it.  Given  the  goal  to  minimize 
the  work  for  both  the  states  and  USDA,  supplying  a  sample  program  to  the  states  is 
attractive. 

We  have  created  a  sample  local  database  in  Microsoft  Access  with  data  input  screens 
and  rudimentary  data  input  checks.  To  make  it  fully  functional,  it  probably  requires 
some  simple  documentation  (under  ten  pages),  some  more  data  input  checks,  some 
simple  reporting  functions,  and  a  simple  file  export  function  (see  below).  The  time 
required  to  add  these  functions  is  probably  less  than  a  month. 

Some  states  will  not  use  a  USDA  database  because  they  have  already  developed  a 
database  with  which  they  are  happy.  For  some  states,  this  database  is  integrated 
with  other  parts  of  their  system  (e.g.,  it  also  holds  financial  data  or  CRE  data). 

For  those  states,  USDA  will  need  to  make  a  policy  choice.  Someone  will  need  to  do 
the  file  translations.  On  whom  does  USDA  want  to  put  the  cost  of  making  the  data 
compatible?  USDA  could  impose  those  costs  on  the  states.  In  that  case,  USDA  could 
publish  specifications  (for  the  names  of  variables  and  the  coding  scheme)  and  then 
require  the  states  to  provide  data  according  to  the  specification.  Presumably,  the 
USDA-provided  program  would  automatically  satisfy  the  specifications,  which  would 
be  another  incentive  for  the  states  to  use  the  USDA  program.  This  approach  seems 
inconsistent  with  the  stated  USDA  desire  to  minimize  the  cost  to  the  states. 

Alternatively,  USDA  could  simply  require  the  states  to  deliver  some  computer  file 
(see  below  for  details).  USDA  would  then  have  to  translate  the  files  itself.  If  USDA 
provides  a  program  that  the  states  can  use  at  their  option,  it  seems  likely  that  many 
(perhaps  most)  states  will  do  so,  thus  the  cost  to  USDA  will  not  be  too  large.  Given 


^See  http://www.conceptual.com/dbmscopy.htm  or  http;//www.stattransfer.com/  for  more  information. 
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USDA’s  stated  desire  to  minimize  the  costs  to  the  states,  it  seems  preferable  for 
USDA  to  offer  to  take  the  data  in  any  well-defined  and  documented  file  format. 

No  State  Database 

Not  all  states  currently  have  a  database,  and  it  is  not  clear  that  it  is  necessary  for  each 
state  to  maintain  a  database.  In  the  Internet  era,  USDA  could  maintain  the  database 
and  states  could  access  it  as  needed  to  generate  reports. 

Under  this  model,  states  would  report  the  results  of  individual  nutrient  analyses  to 
USDA  as  each  review  is  completed.  A  natural  way  to  do  so  would  be  over  the 
Internet.  USDA  would  provide  a  web  site  with  a  data  input  form,  and  the  clerk  in  the 
state  central  office  would  access  the  web  site  and  input  the  data.  We  provide  a  sam¬ 
ple  Microsoft  Front  Page  program  for  the  data  input  at  the  RAND  web  site,^  and  we 
discuss  data  input  and  transmission  in  the  next  section. 

Ideally,  the  nutrient  analysis  software  (e.g.,  NutriKids)  would  generate  a  file  with  all 
the  required  information.  As  we  noted  earlier,  current  software  provides  some,  but 
not  all,  of  the  required  information.  In  that  case,  the  clerk  could  simply  forward  the 
output  from  the  computer  program. 

TRANSMITTING  THE  DATA 

Even  with  agreement  as  to  file  formats,  there  remains  the  question  of  how  the  data 
should  actually  get  to  USDA.  Many  options  are  possible.  At  the  simplest  level,  the 
states  could  send  a  computer  disk  with  the  data  to  USDA  through  the  mad.  The  files 
will  be  quite  small,  so  a  3.5-inch  computer  disk  is  feasible  and  inexpensive,  but  disks 
are  notorious  for  being  corrupted  easily.  There  is  a  danger  of  data  being  sent  in  a 
timely  manner  but  FNS  not  being  able  to  use  them  because  of  corrupt  disks. 

Paper  reports  are  in  principle  possible  but  seem  unnecessarily  cumbersome.  Using 
paper  reports  would  require  USDA  to  rekey  the  information,  which  is  inefficient. 

All  states  reported  that  they  had  (or  would  soon  have)  e-mail  and  Internet  access  for 
their  field  staff  and  for  their  central  office.  E-mail  technology  for  the  transfer  of  com¬ 
puter  files  is  today  quite  robust.  If  states  are  going  to  be  required  to  provide  USDA 


^This  program  is  not  a  complete  prototype  but  demonstrates  the  capability  for  remote  data  entry.  The 
current  program,  however,  does  not  allow  editing  of  already  inputted  data.  That  step  will  be  required  of 
any  production  system.  Errors  in  data  entry  are  not  uncommon  and  they  would  need  to  be  corrected 
(presumably  by  the  state). 

Some  subtle  data  safeguarding  issues  are  raised  by  doing  such  data  editing  over  the  Internet.  Any  Internet 
system  would  require  some  form  of  security  (e.g.,  password  protections  to  prevent  unauthorized  access  to 
or  modification  of  the  data).  Only  USDA  and  the  reporting  state  should  be  able  to  edit  information  for  a 
state — not  other  states  and  not  outsiders. 

Similarly,  USDA  would  need  to  develop  policies  about  who  could  analyze  which  parts  of  the  data.  Clearly, 
USDA  would  want  to  be  able  to  analyze  the  data.  If  USDA  is  maintaining  a  database  of  state  data,  it  seems 
reasonable  that  the  states  should  be  able  to  access,  analyze,  and  download  their  own  data.  USDA  would 
need  to  decide  if  the  filings  were  to  be  considered  public  information  such  that  other  states  or  nongovern¬ 
mental  organizations  would  be  allowed  to  access  and  analyze  (but  not  modify)  the  data.  The  federal 
Freedom  of  Information  Act  may  apply. 
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with  files,  e-mail  is  likely  to  be  the  lowest  cost  and  easiest  option  for  both  USDA  and 
the  states. 

On  the  USDA  side,  the  standard  approach  to  e-mail  submission  is  the  creation  of  a 
dummy  e-mail  account  (perhaps  SMI-input@usda.gov).  States  would  send  their 
submissions  to  this  account.  Whomever  at  USDA  is  currently  responsible  for  the 
data  would  be  given  authority  to  access  the  e-mail  messages.  Such  a  dedicated  ac¬ 
count  would  get  the  state  submissions  out  of  the  stream  of  other  e-mail  coming  to 
the  designated  USDA  staffer.  This  approach  also  allows  USDA  to  change  the  dedi¬ 
cated  staffer  without  informing  or  inconveniencing  the  states. 

Similarly,  if  the  states  provide  data  review-by-review,  input  based  on  Internet  forms 
seems  most  attractive.  All  states  we  talked  to  have  Internet  access,  and  the  software 
could  be  made  easy  to  use.  Input  should  be  completable  in  well  under  fifteen  min¬ 
utes  (perhaps  five  minutes).  States  would  submit  their  data  using  an  Internet  form 
that  looks  similar  to  the  Access  screen  shot  in  Chapter  2.  RAND’s  Survey  Research 
Group  has  used  such  Internet  data  input  for  some  surveys  of  professionals.  Many 
high-tech  marketing  firms  conduct  market  surveys  in  this  way.^  All  e-commerce  sites 
have  a  similar  data  input  operation  (e.g.,  Amazon.com,  eBay.com,  and  eToys.com). 

One  option  for  setting  up  an  Internet  form  is  to  create  the  web  site  (an  HTML  docu¬ 
ment)  with  Microsoft  Access  in  the  background.  This  option  allows  USDA  to  send 
the  data  directly  into  a  Microsoft  Access  database  as  they  are  entered.  However,  this 
has  some  limitations  since  Access  is  not  really  capable  of  handling  multiple  users  at 
the  same  time.  Another  option  is  to  use  an  HTML  document  with  an  application  that 
can  handle  multiple  users,  such  as  Oracle  Personal  Server,  Microsoft  SQL  Server,  or 
Visual  InterDev.  USDA  would  just  need  to  have  the  appropriate  program  loaded  onto 
a  server.  In  all  of  these  cases,  the  web  form  would  look  just  like  a  regular  web  page  to 
the  person  doing  the  data  entry. 

The  only  problem  that  a  web-based  form  might  pose  is  the  currency  of  the  states’ 
web  browsers.  If  FNS  can  be  sure  that  all  the  states  are  using  something  reasonably 
current,  then  web  site  development  will  be  relatively  straightforward.  If  many  states 
are  using  outdated  browsers,  development  might  be  more  complicated.  Depending 
on  how  competent  the  developer  is  (e.g.,  how  well  that  person  understands  the  limi¬ 
tations  of  older  or  obscure  web  browsers),  this  could  have  an  impact  on  the  users. 
However,  this  problem  is  relatively  small  since  many  updated  web  browsers  are 
available  over  the  Internet  for  free.  FNS  would  merely  need  to  instruct  the  states  to 
obtain  a  current  web  browser. 

Such  Internet  input  guarantees  consistent  data  and  minimizes  the  cost  to  USDA  in 
terms  of  tracking  files  submitted  and  reformatting  information.  Furthermore,  the 
system  could  easily  be  constructed  such  that  a  state  could  download  data  to  a  local 
(i.e.,  in  the  home  state)  computer  for  further  analysis  (e.g.,  by  Microsoft  Excel  or  Mi¬ 
crosoft  Access). 


^See  “The  Survey  Says. . .”  in  PC  Magazine,  May  4,  1998,  for  more  information  on  web-based  surveys  and 
using  the  data  in  Microsoft  Access. 


How  Data  Elements  Could  Be  Collected  33 


PUTTING  IT  ALL  TOGETHER 

USDA’s  stated  goal  is  to  collect  the  data  at  minimal  cost  to  the  states.  Therefore,  it 
seems  appropriate  for  USDA  to  accept  the  data  in  any  reasonable  format  that  the 
states  are  willing  to  provide.  Therefore,  we  recommend  that  USDA  accept  data  in  all 
of  the  following  forms: 

1.  USDA  File  Formats:  USDA  develops  and  offers  to  the  states  a  simple  program  al¬ 
lowing  the  creation  and  analysis  of  a  local  state  database.  The  RAND-provided 
Microsoft  Access  program  is  a  start  toward  such  a  program.  States  then  forward 
the  results  annually,  using  the  CRE  deadlines,  to  a  dedicated  e-mail  account  at 
USDA.6 

2.  Other  File  Formats:  USDA  accepts  from  the  states  any  computer  file  they  wish  to 
submit.  The  only  requirements  are  that  the  file  contain  all  of  the  required  data  el¬ 
ements  and  that  the  state  document  the  data  sufficiently  weU  that  USDA  can  read 
the  data  into  its  database.  USDA  would  pay  a  programmer  to  read  the  state  files 
into  its  database.  Given  that  most  states  would  probably  not  choose  this  option, 
the  required  time  would  probably  be  only  a  week  or  two.^  Again,  the  file  would  be 
delivered  annually  by  e-maU  to  a  dedicated  e-mail  account.^ 

3.  Internet  Input:  USDA  develops  an  Internet-based  data  input  program  through 
which  the  states  could  provide  the  results  of  nutrient  analyses  on  a  review-by¬ 
review  basis.  The  Internet  input  option  is  the  least  clear:  Developing  the  software 
will  be  more  expensive  than  the  other  options,  and  it  is  not  clear  whether  the 
states  would  use  the  option. 


CUSTOMIZING  DATABASES  FOR  THE  STATES 

One  issue  that  must  be  considered  when  deciding  the  reporting  system  is  how  the 
states  will  want  to  use  these  data  for  their  own  purposes.  Some  states  already  have  a 
system  in  place  for  reviewing  their  SMI  data.  Others  do  not  have  anything  in  place 
and  do  not  plan  to  develop  a  system  until  they  are  required  to  do  so.  However,  if 
states  are  required  to  report  data  to  FNS,  they  will  undoubtedly  want  to  have  use  of 
that  data  within  the  state  as  well.  Therefore,  this  raises  the  issue  of  ensuring  that  the 
states  are  able  to  customize  the  database  for  their  own  use. 


®To  encourage  states  to  use  USDA’s  program  (and  thus  to  minimize  USDA’s  programming  costs),  USDA 
might  offer  to  pay  for  programming  time  to  read  existing  state  databases  (i.e.,  the  information  in  the 
database)  into  die  USDA  program. 

^In  the  first  year,  the  cost  of  these  two  options — USDA  program  development  for  the  states  to  use  or  state 
data  files  on  receipt — would  be  similar.  Thereafter,  the  first  option  is  likely  to  be  cheaper.  If  states  convert 
to  the  USDA  program,  then  it  is  easy  for  USDA  to  change  the  submission  rules  (e.g.,  to  add  new  fields,  to 
change  the  codes,  to  improve  the  data  validadon).  However,  if  the  states  customize  USDA’s  program, 
USDA  revisions  to  its  program  may  require  additional  work  by  the  states  to  redo  their  customizations. 
Because  of  this  and  to  minimize  the  need  for  customization,  USDA  should  poll  the  states  about  additional 
program  features  desired. 

®So  that  USDA  would  not  have  to  do  conversions  more  than  once  a  year,  the  information  should  be  sent 
annually. 
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If  states  already  have  their  own  system  and  they  will  be  sending  their  data  to  FNS  in 
their  current  application  (i.e.,  Microsoft  Access  or  Excel,  Oracle),  then  they  can  easUy 
customize  the  system  to  include  other  data  elements  for  their  internal  analyses.  They 
would  then  just  need  to  send  the  required  data  elements  to  FNS. 

If  states  were  to  use  a  protocol  given  to  them  by  FNS  (e.g.,  the  Access  database  RAND 
has  designed),  states  could  customize  the  system  in  much  the  same  way  they  would 
their  own  system.  Again,  they  would  just  need  to  send  the  appropriate  data  elements 
to  FNS. 

In  the  case  of  a  web-based  system,  two  issues  pose  difficulty  for  customization.  The 
first  concerns  getting  their  own  data  without  others  also  having  access  to  it.  Allowing 
states  to  query  any  data  in  a  web-based  system  is  not  difficult,  but  allowing  states  to 
access  only  their  own  data  is  probably  more  desirable  but  requires  some  program¬ 
ming.  The  second  issue  is  actual  customization.  If  states  were  able  to  download  a 
plain  text  file  from  a  web-based  system,  they  could  potentially  merge  it  with  the 
other  data  elements  that  they  want  to  add.  However,  this  requires  an  identifier  for  the 
review  to  have  something  on  which  to  merge.  It  is  clear  that  the  web-based  system 
would  be  the  most  difficult  for  states  to  customize.  However,  some  states  that  do  not 
currently  have  systems  in  place  may  wish  to  use  this  since  they  may  not  see  a  need 
for  additional  analyses  using  other  data  elements.  A  web-based  system  would  save 
them  having  to  develop  or  maintain  anything  new. 

MANAGING  THE  DATABASE  AT  USDA 

We  conclude  with  some  thoughts  about  the  database  at  USDA.  There  are  about 
20,000  SFAs  in  the  country.  With  a  five-year  review  cycle  and  some  SFAs  using  mul¬ 
tiple  menu  planning  systems,  USDA  would  expect  on  the  order  of  4,000  new  records 
per  year.  Each  record  would  have  between  20  and  30  data  elements. 

This  is  a  moderate-sized  database  that  can  be  run  on  the  standard  desktop  personal 
computer  purchased  today  (late  1999).^  Clearly,  e-mail  and  web  access  would  be 
needed.  For  software,  USDA  could  use  Microsoft  Access,  an  Oracle  database,  or  a 
more  conventional  statistical  package  such  as  SAS. 

The  tabulations  required  by  USDA  are  straightforward  and  could  be  precoded.  In  the 
language  of  SAS,  they  are  “frequencies”  with  “subsetting  if’  and  “by”  statements. 
The  systems  should  probably  include  the  possibility  of  ad  hoc  queries.  QBE  (Query 
by  Example)  is  the  easiest  way  to  specify  such  queries.  An  analyst  with  rudimentary 
experience  in  statistical  analysis  (e.g.,  those  required  to  complete  a  masters  program 
in  public  policy)  should  be  able  to  program  the  required  queries  without  major  assis¬ 
tance. 


®One  example  is  Dell’s  low-level  Dimension  L433CD  with  a  433  MHz  Celeron  processor,  64MB  RAM,  and 
4.3GB  hard  disk.  List  price  is  $1,299  as  of  November  1999. 


Chapter  Five 

DISCUSSION  OF  THE  CONTENT  OF  SPECIFIC  MEALS 


The  discussion  of  the  previous  four  chapters  has  considered  the  technical  question  of 
what  information  USDA  should  collect  about  SFA-level  SMI  reviews  and  how  that 
information  should  be  forwarded  to  USDA.  In  this  chapter,  we  briefly  consider  the 
broader  issues  of  how  the  SMI  effort  in  general  (especially,  state  reporting  to  USDA) 
advances  the  statutory  goals  of  the  NSLP  (in  particular,  the  compliance  of  NSLP 
meals  with  the  DGA). 

We  begin  with  a  discussion  of  two  positive  effects  of  the  current  review  procedures: 
First,  the  experience  of  preparing  for  a  review  itself  raises  consciousness  of  nutri¬ 
tional  issues;  and  second,  as  intended,  the  results  of  the  review  lead  to  improvements 
in  the  nutritional  content  of  meals. 

We  then  consider  three  specific  concerns  about  the  quality  of  the  data  that  are  col¬ 
lected.  Combined  with  the  required  effort,  these  concerns  lead  some  state  officials  to 
question  the  value  of  applying  limited  staff  to  the  SMI  effort.  For  example.  New  York 
state  staff  commented  that  “too  much  of  [our]  resources  are  directed  toward  artificial 
data  gathering,  rather  than  feeding  kids.” 

In  particular,  three  considerations  suggest  that  the  present  SMI  review  program  may 
not  be  ideal  for  achieving  the  goals  set  out  in  the  USDA  Strategic  Plan — especially 
that  of  ensuring  that  school  meals  are  consistent  with  the  Dietary  Guidelines  for 
Americans  and  the  Recommended  Daily  Allowances.  The  first  issue  concerns  the 
quality  of  the  review  information  for  the  meals  reviewed.  The  second  issue  concerns 
the  representativeness  of  the  meals  being  reviewed  at  the  SFA  level  (i.e.,  how  useful  is 
the  information  for  parents  in  a  given  SFA  about  the  meals  being  served  by  their 
SFA).  The  third  issue  concerns  the  representativeness  of  meals  being  reviewed  at  the 
state  level  (i.e.,  how  useful  would  some  average  of  information  about  all  meals  across 
groups  of  schools  be  to  USDA).  We  discuss  each  of  these  considerations  and  suggest 
additional  steps  the  USDA  could  take  to  address  them. 

PREPARING  FOR  THE  REVIEW 

Our  interviews  in  the  states  emphasized  that  preparing  for  the  review  itself  often 
generates  positive  spillovers.  Several  states  ran  seminars,  attended  by  representa¬ 
tives  from  large  numbers  of  SFAs,  to  help  them  prepare  for  their  SMI  reviews.  The 
impending  SMI  reviews  forced  SFA  staff  to  focus,  in  general,  on  nutritional  issues 
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and,  in  particular,  on  the  DGA,  and  considerable  nutritional  education  of  SFA  staff 
appears  to  have  occurred. 

Furthermore,  preparing  for  the  reviews  itself  probably  improved  procedures  and 
consciousness  of  nutritional  issues.  In  some  SFAs,  the  SMI  requirement  induced 
a  movement  to  nutrient-based  menu  planning  systems.  Even  in  the  majority  of  SFAs 
that  have  not  made  the  move  to  nutrient-based  menu  planning  systems,  preparing 
for  the  SMI  reviews  forced  changes  in  procedures.  Detailed  written  menus  needed 
to  be  developed.  Detailed  nutritional  information  needed  to  be  obtained  from 
suppliers. 

These  efforts  themselves  are  likely  to  result  in  improved  nutritional  content  of  meals 
served.  Proper  menus  and  nutritional  information  for  ingredients  are  prerequisites 
for  nutritional  improvement.  The  increased  attention  to  the  nutritional  content  of 
meals  is  itself  likely  to  have  positive  effects  on  the  nutritional  content  of  meals  served 
in  the  future. 

RESULTS  OF  THE  REVIEW 

The  primary  mechanism  through  which  reviews  are  likely  to  affect  future  meals 
served  is  through  the  results  of  the  reviews.  First,  state  officials  reported  that  both 
they  and  SFA  officials  were  often  surprised  by  the  results  of  the  reviews.  Formal 
nutrient  analysis  often  yielded  results  very  different  than  those  expected  by  trained 
nutritionists  conducting  informal  reviews  of  menus  and  procedures.  State  officials 
reported  that  on  seeing  review  results,  SFA  officials  often  resolved  to  improve  the 
nutritional  content  of  their  meals. 

Furthermore,  the  reported  formal  “Correction  Plan”  is  specifically  intended  to  im¬ 
prove  the  nutritional  content  of  meals  served.  State  officials  meet  with  SFA  officials 
to  consider  the  results  of  the  SMI  review,  to  specify  changes  in  SFA  policy  (e.g., 
menus,  ingredients,  suppliers,  cooking  procedures)  to  improve  the  nutritional  con¬ 
tent  of  meals  served,  and,  ideally,  to  bring  them  into  compliance  with  DGA. 

The  interval  between  reviews  is  long  enough — about  five  years — that  improvements 
in  nutrition  are  not  immediately  necessary.  Failure  to  change  procedures  as 
specified  in  the  “Correction  Plan”  will  probably  not  be  detected,  yet  willful 
noncompliance  seems  unlikely.  Almost  everyone  in  the  system  would  like  to 
improve  the  nutritional  content  of  meals.  However,  staff  changes,  changes  in  menus 
and  the  nutritional  content  of  ingredients,  and  different  priorities  (e.g.,  cost,  staffing) 
compete  for  the  attention  of  SFA  management.  Despite  these  concerns,  it  seems 
likely  that  the  results  of  the  reviews  and  the  “Correction  Plan”  itself  will  improve  the 
quality  of  future  meals  served. 

QUALITY  OF  THE  REVIEW  INFORMATION 

Granted  these  positive  effects  of  preparing  for  the  SFA  reviews  and  of  changed  pro¬ 
cedures  in  response  to  the  results  of  these  reviews,  the  primary  focus  of  this  report 
has  been  on  procedures  for  collecting  the  results  of  the  reviews  and  forwarding  them 
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to  USDA.  For  that  purpose,  SMI  reviews  are  only  as  good  as  the  quality  of  underlying 
nutrient  analysis.  However,  staff  in  many  states  questioned  the  quality  of  the  re¬ 
views.  In  particular,  they  stated  that  the  results  of  reviews  would  vary  depending  on 
who  did  the  review. 

Such  concerns  about  inter-rater  reliability  are  common  in  such  programs  (e.g.,  SSA 
disability  reviews;  see  Parsons,  1991,  and  the  literature  it  spawned)  and  are  likely  to 
be  particularly  salient  in  a  new  program  such  as  SMI.  It  is  usually  nearly  impossible 
to  eliminate  such  reliability  problems.  Nevertheless,  standard  steps  exist  to  mini¬ 
mize  reliability  problems.  Such  efforts  usually  begin  with  the  promulgation  of  com¬ 
mon  procedures  for  how  to  do  a  review  and  how  to  handle  standard  problems.  They 
then  proceed  to  formal  initial  and  ongoing  training  in  the  common  procedures. 
Finally,  a  program  to  test  the  quality  of  the  reviews  can  be  put  in  place  (i.e.,  re¬ 
reviewing  a  sample  of  completed  reviews).  Depending  on  the  importance  and  avail¬ 
able  funding,  initiatives  to  address  concerns  about  inter-rater  variation  may  be  worth 
pursuing. 

REPRESENTATIVENESS  OF  REVIEW  AT  THE  SEA  LEVEL 

Even  if  it  were  assumed  that  the  reviews  are  perfect,  it  is  not  clear  how  informative 
the  results  of  a  review  (of  meals  served  one  given  week  in  a  given  school)  are  about  all 
the  meals  served  in  that  school  (or  SFA).  For  example,  if  the  reviewed  meals  meet  the 
guidelines,  does  that  imply  that  all  meals  served  by  the  SFA  meet  the  guidelines? 
Conversely,  if  the  reviewed  meals  do  not  meet  the  guidelines,  does  that  imply  that  no 
meals  meet  the  guidelines?  The  distinction  is  important.  Nutrition  is  a  result  of  all 
meals  served,  not  the  meals  that  were  reviewed.  Put  differently,  if  the  results  of  an 
SFA’s  review  were  to  become  publicly  available  at  an  SFA  level,  is  it  reasonable  for 
parents  to  complain?  Or  rest  assured? 

The  minimum  required  SMI  effort  reviews  only  a  very  small  sample  of  meals  served, 
the  reviews  take  place  only  every  five  years,  and  only  one  school  in  each  selected  SFA 
is  reviewed  for  each  menu  plan  and  age  group.  Within  that  school,  only  one  week’s 
meals  are  reviewed.  The  choice  of  schools  and  weeks  is  not  random  but  rather  is 
done  at  the  convenience  of  the  reviewers.  There  are  serious  questions  about  the 
usefulness  of  this  information.  If  every  school  in  an  SFA  always  served  the  same 
meals  all  the  time,  in  exactly  the  same  way,  using  exactly  the  same  ingredients  and 
suppliers,  then  one  SMI  review  would  be  enough.  That  level  of  uniformity  is,  of 
course,  not  present.  Instead,  it  is  quite  plausible  that  the  variation — across  years, 
across  schools,  across  weeks — is  so  large  that  the  results  of  one  review  provide  little 
information  about  the  nutritional  content  of  the  meals  consumed  in  that  district.  If 
so,  then  any  SFA-level  reports  to  USDA  have  litde  information  at  the  SFA-level  and 
are  thus  not  useful  for  parents. 

The  amount  of  variation — across  years,  across  schools,  across  weeks — and  therefore 
the  usefulness  of  SFA-level  data  is,  in  principle,  an  empirical  question.  To  address 
this  issue,  USDA  could  fund  a  small  pilot  study.  Such  a  pilot  study  would  randomly 
select  a  small  group  of  SFAs  (perhaps  a  few  dozen).  In  the  selected  SFAs,  the  pilot 
study  would  collect  SMI  type  information  across  schools  and  through  time.  From 
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these  reviews,  it  would  be  possible  to  characterize  the  ability  of  an  individual  SMI 
review  to  provide  information  about  the  nutritional  content  of  all  meals  provided  by 
the  SFA. 

REPRESENTATIVENESS  OF  REVIEW  AT  THE  STATE  LEVEL 

The  USDA  Strategic  Plan  requires  information  about  all  meals  served.  For  the  pre¬ 
sent  purposes,  USDA  is  only  seeking  information  on  the  meals  that  are  reviewed. 
Review  processes  can  only  collect  information  about  meals  reviewed.  Reviews  are 
expensive:  reviewing  a  large  number  of  meals  could  be  very  expensive.  However, 
under  plausible  assumptions  and  procedures,  reviews  of  a  small  number  of  meals  are 
informative  about  all  meals  served. 

If  each  review  is  properly  done,  then  standard  arguments  from  statistical  sampling 
imply  that  we  can  construct  a  statistically  accurate  portrait  of  all  meals  served  by  a 
group  of  SFAs  (e.g.,  a  state)  by  averaging  results  across  multiple  reviews  and  across 
individual  weeks  from  individual  schools  from  multiple  districts.  This  would  be  true 
despite  the  fact  that  no  individual  review  provides  a  complete  characterization  of  the 
meals  provided  by  an  SFA. 

For  this  standard  statistical  sampling  argument  to  apply,  however,  it  is  necessary  for 
standard  random  sampling  assumptions  to  apply.  For  this  situation  that  means: 

1.  Random  assignment  of  which  SFAs  are  to  be  reviewed  in  a  given  year  (e.g.,  per¬ 
haps  random  assignment  to  each  of  the  five  years  of  the  review  cycle). 

2.  Random  assignment  of  which  schools  within  the  SFA  are  to  be  reviewed;  other¬ 
wise,  SFAs  might  pick  the  schools  with  the  “best”  nutritional  content. 

3.  Random  assignment  of  which  week  during  the  school  year  reviews  might  occur; 
otherwise,  the  timing  of  the  review  might  be  correlated  with  particularly  good  or 
bad  meals  (e.g.,  around  holidays  or  the  beginning  or  ending  of  the  school  year). 

Furthermore,  the  review  week  must  be  a  surprise.  Otherwise,  an  SFA  might  substi¬ 
tute  a  “better”  (more  nutritious)  menu  for  the  review  week. 

In  addition,  if  USDA’s  goal  is  to  construct  a  statistically  precise  estimate  of  all  meals 
served,  then  standard  sampling  arguments  suggest  a  different  review  schedule. 
Current  regulations  require  one  review  per  SFA  in  each  five-year  cycle,  regardless  of 
the  size  of  the  district.  SFAs,  however,  vary  greatly  in  their  size.  At  the  extreme,  about 
half  of  the  meals  in  New  York  State  are  served  by  the  New  York  City  public  school 
system.  Statistical  considerations  suggest  allocating  more  resources  (i.e.,  more  re¬ 
views,  perhaps  more  than  every  five  years)  to  such  large  SFAs,  and  fewer  resources 
(i.e.,  fewer  reviews,  perhaps  less  frequendy  than  every  five  years)  to  smaller  SFAs. 
Estimating  how  much  more  would  require  the  results  of  a  pilot  study  like  the  one  dis¬ 
cussed  above. 

Such  an  approach  has  a  precedent:  Current  CRE  regulations  already  promote  more 
intensive  efforts  in  larger  districts.  Under  the  Expanded  Review  Cycle,  “State 
Agencies  are  encouraged  to  conduct  administrative  reviews  of  large  school  food  au- 
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thorities  and  of  any  school  food  authorities  which  may  benefit  from  a  more  frequent 
interval  than  the  5-year  cycle  required  by  paragraph  (c)(1). 

Adjusting  the  intensity  of  the  review  schedule  with  the  size  of  the  SFA  is  also  attrac¬ 
tive  from  a  burden  perspective.  Many  state  officials  expressed  concerns  about  the 
burden  of  SMI  reviews  for  small  SFAs.  For  small  SFAs,  food  service  is  a  small-scale 
operation.  Nebraska  suggested  that  FNS  should  accept  data  aggregated  by  school 
size,  since  they  have  so  many  very  small  SFAs.  In  Texas,  50  percent  of  the  students 
are  in  46  SFAs  and  the  other  50  percent  are  spread  out  over  more  than  1,000  SFAs. 
According  to  the  state  staff,  many  of  these  small  SFAs  feel  that  SMI  is  beyond  them 
because  they  do  not  have  the  necessary  education  or  expertise.  In  most  small  SFAs, 
staff  have  little  nutritional  training,  and  the  cook  is  often  the  only  food  service  staff 
person.  In  such  situations,  SMI  reviews  are  a  great  burden.^  State  officials  expressed 
concern  that  SFAs  would  pull  out  of  the  program  once  they  knew  FNS  would  require 
the  collection  of  nutritional  content. 

USDA  has  already  partially  moved  toward  lessening  this  burden  by  allowing  less- 
intensive  review  of  RCCIs.^  Careful  consideration  of  less-intensive  review  for  small 
SFAs  might  be  worthwhile.  Exempting  small  SFAs — under  1,000  students — would 
exclude  43  percent  of  the  SFAs,  but  only  5  percent  of  the  students  from  SMI.^ 
Eliminating  (or  at  least  scaling  back)  review  of  these  small  SFAs  might  lower  resis¬ 
tance  to  the  program  and  the  burden  of  the  unfunded  mandate  on  the  state  authori¬ 
ties.  The  cost  in  the  completeness  of  the  data  would  be  small. 

CONCLUSION 

This  final  chapter  has  reviewed  broader  issues  relating  to  the  SMI  program.  We  dis¬ 
cussed  the  positive  effects  of  SMI  reviews  arising  from  efforts  to  prepare  for  the  re¬ 
view  and  from  changes  in  meals  due  to  the  results  of  the  review. 

This  report  focused  on  how  states  should  report  the  results  of  SMI  reviews  to  USDA. 
For  that  purpose,  the  quality  and  representativeness  of  the  underlying  review  results 
are  crucial.  We  presented  three  concerns  about  current  procedures  and  suggested 
some  actions  USDA  could  undertake  to  address  these  concerns. 

USDA  is  aware  of  these  concerns,  and  similar  issues  arise  in  the  CRE  process.  The 
SMI  review  process  is  still  in  its  first  cycle.  Now,  as  the  first  cycle  winds  down  and  as 
USDA  considers  how  states  should  report  SMI  results,  is  an  appropriate  time  to  con¬ 
sider  the  research  activities  and  operational  changes  discussed  in  this  chapter. 

However,  it  is  important  to  note  the  concerns  expressed  by  the  states  about  the  effort 
required  to  perform  the  SMI  reviews.  How  to  proceed  with  the  review  program  re- 


^Paragraph  (cKD  states  that  ‘‘at  a  minimum,  State  Agencies  shall  conduct  administrative  reviews  of  all 
school  food  authorities  at  least  once  during  each  5-year  review  cycle,"  210  USC  18. 

^These  might  also  be  the  SFAs  whose  meals  have  particularly  poor  nutritional  content. 

^FNS  Memorandum  #99-5;  1999. 

^The  Gallup  Organization  and  PROMAR  International,  1999. 
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quires  balancing  the  direct  effects  on  the  nutritional  content  of  meals,  the  utility  of 
the  aggregated  data  (given  the  quality  of  the  underlying  reviews),  and  the  statutory 
requirement  against  the  required  effort  by  state  and  SFA  staff.  Staff  in  several  states 
argued  strongly  that  they  did  not  believe  the  effort  was  worthwhile  given  the  per¬ 
ceived  benefit. 


Appendix  A 

PROTOCOL  FOR  DATA  COLLECTION 


Date:  _ 

Name:  _ 

Agency:  _ 

Title:  _ 

Phone  Number: 

Name:  _ 

Agency:  _ 

Title:  _ 

Phone  Number: 

Name:  _ 

Agency:  _ 

Title:  _ 

Phone  Number: 

Name:  _ 

Agency:  _ 

Title:  _ 

Phone  Number: 
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FNS  hopes  to  ultimately  be  able  to  aggregate  nutrient  analysis  information  to  the 
state  and  national  level.  To  do  this,  it  needs  certain  data  elements,  and  these  data 
elements  have  to  be  in  electronic  format  for  each  School  Food  Authority  (SFA)  in  the 
state.  This  project  will  look  at  what  seven  states  are  currently  doing  and  what  data 
elements  are  being  collected.  FNS  does  not  want  us  to  recommend  major  changes  to 
the  way  that  your  state  is  currently  collecting  the  SMI  information.  Instead,  we  wiU 
be  looking  at  what  you  are  doing  and  what  minor  changes  you  could  make  in  order  to 
get  FNS  what  it  needs. 

RAND  plans  to  adhere  to  the  following  timeline  for  this  project.  As  you  can  see,  there 
is  quite  a  bit  of  time  for  comments  from  FNS.  FNS  plans  to  share  the  preliminary 
results  with  the  states  and  give  the  states  an  opportunity  to  comment  on  the  findings 
before  RAND  completes  any  of  the  memos  or  reports. 


TIMELINE  FOR  USDA  CNAP  TASK  #4 


School  Meal  Program  Government  Performance  and  Results  Act  of  1993 

Standards 


Item /Deliverable 

Tentative  Date 

Data  collection  protocols  for  state  visits  finalized 

5/21/99 

Fieldwork  begins 

6/1/99 

Fieldwork  ends 

9/1/99 

First  memo  draft  delivered  to  FNS 

10/1/99 

FNS  provides  comments  to  RAND 

10/22/99 

First  memo  revised  delivered  to  FNS 

11/5/99 

FNS  provides  comments  to  RAND 

11/26/99 

Second  memo  draft  delivered  to  FNS 

12/24/99 

FNS  provides  comments  to  RAND 

1/21/00 

Second  memo  revised  delivered  to  FNS 

2/4/00 

FNS  provides  comments  to  RAND 

2/25/00 

Report  draft  delivered  to  FNS 

3/24/00 

FNS  provides  comments  to  RAND 

5/19/00 

Final  report  text  delivered  to  FNS 

6/9/00 

Final  report  documents  delivered  to  FNS 

8/11/00 
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BACKGROUND  QUESTIONS 

1.  How  many  SFAs  are  there  in  your  state? 

_  SFAs  DK 

2.  How  do  you  define  an  SFA?  Is  it  a  school  district  or  something  else? 


3.  When  did  your  state  begin  SMI  reviews? 

1996/1997  1997/1998  1998/1999  Haven’t  started  DK 

4.  Why  did  you  start  then  as  opposed  to  earlier  or  later? 


How  many  SMI  reviews  were  completed  in 

1996/1997? 

Reviews 

DK 

1997/1998? 

Reviews 

DK 

1998/1999? 

Reviews 

DK 

How  do  you  define 

a  completed  review? 

7.  When  do  you  expect  to  complete  the  first  round  of  SMI  reviews? 

1998/1999  1999/2000  2000/2001 

2001/2002  2002/2003  DK 

8.  Do  you  think  your  state  will  need  to  make  any  changes  in  the  future  to  process 
or  staffing  in  order  to  complete  the  SMI  reviews  in  5  years? 


9.  Are  SMI  reviews  done  in  conjunction  with  CRE  reviews? 

Yes  No  DK 

10.  If  you  do  SMI  reviews  in  conjunction  with  CRE,  did  you  have  to  add  staff  to  do 
this?  What  kind  of  training  was  involved? 
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11.  Do  your  reviewers  have  access  to  e-mail? 


Yes 


No 


DK 


12.  Do  your  reviewers  have  access  to  the  Internet? 


Yes 


No 


DK 


13.  Is  there  any  other  software  (i.e.,  MS  Excel)  that  reviewers  use  for  completing 
reviews?  If  so,  what?  Is  the  same  software  used  at  the  state  level  and  at  the  SEA 
level? 


Yes 


No 


DK 


14.  How  many  people  are  involved  in  doing  SMI  reviews  and  analysis? 

_  People 

15.  Where  are  they  located?  For  whom  do  they  work? 


16.  What  are  each  of  their  roles  in  the  SMI  reviews? 


17.  Does  the  state  agency  have  access  to  the  nutrient  analysis  information? 

Yes  No  DK 

18.  Do  you  feel  that  the  SMI  reviews  are  necessary  to  bring  school  meals  into 
compliance  with  the  Dietary  Guidelines  for  Americans  and  the  Recommended 
Daily  Allowances? 

Yes  No  DK 


Comments: 
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Do  you  think  it  would  be  difficult  for  the  reviewers  in  this  state  to  provide 
information  to  FNS  directly? 

Yes  No  DK 


Comments: 
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PROCESS  IN  THE  STATE 

We  are  trying  to  create  a  flowchart  that  shows  how  and  where  the  information  is 
collected,  whom  the  information  is  passed  on  to,  when  and  how  the  information  is 
revised,  and  what  format  the  information  is  in  at  every  stage.  I  would  like  to  go 
through  the  process  separately  for  an  SFA  using  a  food-based  system  and  an  SFA 
using  a  nutrient-based  system,  since  these  processes  are  likely  to  be  different.  Bear 
in  mind  that  FNS  will  require  you  to  provide  the  data  it  requests  for  both  food-based 
and  nutrient-based  systems. 

1.  Who  collects  the  raw  information  for  nutrient  analyses  on  food  offered  in  any 
given  SFA? 


2.  With  what  organization  is  this  person  employed? 


3.  In  what  format  is  the  information  collected?  Does  this  format  change  over  the 
course  of  the  review?  For  example,  if  the  initial  information  is  collected  on  hard 
copy,  is  it  ever  converted  to  an  electronic  version? 


4.  Who  performs  the  nutrient  analysis  of  this  information? 


5.  Are  there  any  steps  between  the  initial  collection  of  information  and  the 
nutrient  analysis?  If  so,  what  are  they  and  who  performs  them? 
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6.  Is  the  nutrient  analysis  ever  revised  after  it  is  initially  performed?  If  so,  when 
and  by  whom?  Where  is  the  revision  information  recorded? 


7.  How  often  are  nutrient  analyses  usually  revised? 


8.  Are  data  elements  ever  added  or  deleted  from  the  information  during  this 
process? 


9.  Is  the  information  aggregated  in  some  way  other  than  at  the  state  level?  For 
example,  at  a  district  or  regional  level?  If  so,  at  what  level? 


10.  Where  are  SFA-level  records  kept  and  in  what  format? 


11.  When  is  the  information  sent  to  the  state  and  by  whom? 


12.  How  is  the  information  sent  to  the  state?  Electronically?  Hard  copy?  If 
electronically,  please  describe  the  protocols  used.  For  example,  is  the 
information  recorded  on  a  diskette?  Sent  by  e-mail?  Other? 
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13.  Where  is  the  information  kept  and  in  what  format?  If  applicable,  please  give  a 
name  and  telephone  number  for  the  person  who  would  have  this  information  at 
the  state  level. 


14.  Does  the  state  do  an  independent  nutrient  analysis  for  SMI  reviews,  or  does  the 
state  review  existing  analyses,  or  both?  In  which  cases  does  the  state  do 
independent  reviews?  In  which  cases  does  the  state  review  nutrient  analyses 
performed  elsewhere?  (Keep  in  mind  that  it  is  possible  for  the  state  to  do  both  if 
it  reviews  nutrient  analyses  done  by  an  SFA  using  a  NuMenus  or  Assisted 
NuMenus  system  but  does  the  actual  nutrient  analyses  for  SFAs  using  food- 
based  systems.) 


15.  At  what  point  would  it  be  best  to  have  the  nutrient  analysis  information  sent  to 
FNS?  From  the  state?  From  the  reviewers  themselves?  Why? 


16.  Which  of  these  steps,  if  any,  would  need  to  change  in  order  to  meet  FNS’s  goals 
for  the  selected  data  elements  being  sent  in  electronic  format? 


17.  It  is  possible  that,  in  the  future,  FNS  may  be  able  to  negotiate  with  the 
companies  that  have  created  the  nutrient  analysis  software  to  add  a  function 
where  you  would  be  able  to  create  the  report  for  FNS  right  from  the  software.  If 
that  were  to  happen,  what  changes  would  you  need  to  make  to  your  current 
procedure? 
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18.  Which  of  the  required  data  elements  are  currently  missing  from  the  software 
package  you  are  using  for  your  nutrient  analyses? 


19.  If  the  software  companies  do  not  agree  to  add  this  function,  how  will  you 
incorporate  the  additional  elements  into  an  electronic  report  for  FNS? 


20.  Do  you  think  that  there  are  any  data  elements  that  should  be  added  to  or 
deleted  from  the  list  to  send  to  FNS?  If  so,  which  items  and  why? 


21.  Do  you  have  any  opinion  on  sending  the  information  to  FNS  electronically? 
Will  this  create  any  problems  for  your  state? 


22.  How  often  do  you  think  the  states  should  have  to  report  this  information  to 
FNS?  FNS  is  required  to  prepare  an  annual  strategic  plan.  Therefore,  it  is 
leaning  toward  annual  collection  of  this  information.  Would  this  cause 
problems  for  your  state? 


23.  Can  you  think  of  any  alternatives  for  any  of  the  processes  we  have  discussed  so 
far? 
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24.  FNS  would  like  us  to  solicit  comments  from  the  state  about  this  process.  Do  you 
have  anything  that  you  would  like  us  to  pass  on  to  them? 
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DATA  ELEMENTS 

Based  on  conversations  with  FNS,  the  following  data  elements  will  be  necessary  in 
order  to  aggregate  the  information  at  the  state  or  national  level  in  a  way  that  will  be 
useful.  The  data  elements  will  include  both  the  measures  of  tlie  eight  quantitative 
and  three  qualitative  standards  for  each  SFA,  as  well  as  the  standards  themselves. 
These  standards  are  necessary  because  FNS  may  not  have  access  to  the  standards  an 
SFA  is  using,  especially  if  it  is  using  a  nontraditional  USDA-approved  system. 

In  the  “Type”  column,  C  indicates  that  a  character  response  is  required;  N  indicates 
that  a  numeric  response  is  required. 


Element _ 

Unique  reference  code 
for  analysis 


Name  of  SFA 
Contact  Person  for 
Analysis 

Contact  Person  Phone  # 
Contact  Person  E-mail 
Contact  Person  Address, 
line  1 

Contact  Person  Address, 
line  2 

Contact  Person  Address, 
line  3 

Contact  Person  City 
Contact  Person  State 
Contact  Person  Zip  Code 
Name  of  Person  Doing 
Review 

Name  of  Person  Doing 
Analysis 

Start  Date  of  Review 
Week 

End  Date  of  Review  Week 

Date  Analysis  Closed 

State  in  Which  Review 
Was  Done 
Type  of  School  Food 
Authority 


Number  of  Schools  in 
SFA 


Reporting  Format 


J  (First) 
.1  (Last) 


|_U/U_|/|_I_U_I 

(MM  /DD/YYYY) 

UU/U_|/|_U_U 

(MM/DD/YYYY) 

I _ I _ I  (Month) 

(Year) 


Type _ Notes _ 

C  Unique  code  for 
FNS;  State  wiU 
not  repeat 
reference  code 
for  any  future  or 
past  analysis 
C 
C 

C 

N 

C 

C 

c 

c 


c 

N 

c 

c 

N 

N 

N 

N 

C 

N  l=Public  School 
District 

2=Private  School  or 
District 

3=Residential  ChUd 
Care  Institution 
4=Other 
N 
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Menu  Planning  System 
Used 


Clarification  of  Other 
USDA-Approved 
System 

Was  Analysis  for  Lunch 
Only  or  Lunch  and 
Breakfast? 

Analysis  Weighted? 

Was  a  Plan  of  Correction 
Needed? 

Lowest  Grade /Age  Range 
Used  for  Analysis 


(free  form) 


Highest  Grade /Age 
Range  Used  for 
Analysis 


Analysis  Done  by  Grade 
Range  or  Age  Range 
‘‘Average  Daily  Number 
of  Lunches  Served 
During  Week  for 
Which  Analysis  Done 
Actual  Content  from 
Analysis  Software 
Output: 

Caloric  content  (kcal) 

Fat  content  (g) 

Saturated  Fat  content  (g) 
Protein  content  (g) 
Calcium  content  (mg) 
Iron  content  (mg) 
Vitamin  A  content  (RE) 
Vitamin  C  content  (mg) 
Cholesterol  content  (mg) 


C  N  =  NuMenus 

A  =  Assisted 
NuMenus 
T  =  Traditional 
Food-Based 
E  =  Enhanced 
Food-Based 
O  =  Other  USDA- 
Approved 
System 

C  Define  Other 

USDA-  Approved 
System(s)  Used 
C  L  =  Lunch  Only 

B  =  Both  Lunch  & 
Breakfast 

C  Y  =  Yes,  N  =  No 

N  0=No 

l=Yes 

C  P  =  Pre- 

Kindergarten/ 
Invalid  Response 
K  =  Kindergarten/ 
Invalid  Response 
0  =  Invalid 

Response /Infant 

1  =  First  Grade /1st 
Year 

2  =  Second 

Grade/2nd  Year 
(and  so  on) 

C  P  =  Pre- 

Kindergarten/ 
Invalid  Response 
K  =  Kindergarten/ 
Invalid  Response 
0  =  Invalid 

Response/Infant 

1  =  First  Grade/ 1st 
Year 

2  =  Second 

Grade/2nd  Year 
(and  so  on) 

C  G  =  Grade  Range 
A  =  Age  Range 
N 


N 

N 

N 

N 

N 

N 

N 

N 

N 
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Sodium  content  (mg) 

l_U_UI 

N 

Fiber  content  (g) 

Standards  for  Menu  Plan 

UUI  •  l_l_l 

N 

Used: 

Caloric  content  (kcal) 

l-l_l_ui 

N 

Fat  content  (g) 

UUI  •  UUI 

N 

Saturated  fat  content  (g) 

UUI  •  UUI 

N 

Protein  content  (g) 

UUI  ■  UUI 

N 

Calcium  content  (mg) 

UUUI 

N 

Iron  content  (mg) 

UUI  •  UUI 

N 

Vitamin  A  content  (RE) 

UUUUI 

N 

Vitamin  C  content  (mg) 

UUI  •  UUI 

N 

Comments 

(free  form) 

c 

Space  for  any 

clarification 
comments  from 
the  SFA 

^  — 


Appendix  B 

INDIVIDUALS  INTERVIEWED  IN  THE  SEVEN  STATES 


Table  B.l 

Individuals  Interviewed  in  California 


Individual 

Interviewed 

Agency 

Title 

Jean  Naylor 

State  Department  of  Education,  Field 
Services  Unit 

Child  Nutrition 
Supervisor  II 

Louise  Casias 

State  Department  of  Education,  Field 
Services  Unit 

Child  Nutrition 
Supervisor  I 

Violet  Henry 

State  Department  of  Education,  Field 
Services  Unit 

ChUd  Supervisor  1 

Valerie  Fong 

State  Department  of  Education,  School 
Nutrition  Programs  Unit 

Staff  Services 
Manager  II 

Jan  Bamhouse 

State  Department  of  Education,  School 
Nutrition  Standards  Unit 

ChUd  Nutrition 
Supervisor  I 

Andrew  Laufer 

State  Department  of  Education,  Field 
Services  Unit 

Child  Nutrition 
Supervisor  I 

Cindy  Schneider 

State  Department  of  Education,  School 
Nutrition  Standards  Unit 

Child  Nutrition 
Consultant 

Marilyn  Briggs 

State  Department  of  Education,  School 
Nutrition  Services  Division 

Director 

Table  B.2 

Individuals  Interviewed  in  Georgia 

Individual 

Interviewed 

Agency 

Title 

Annette  Bomar 
Hopgood 

Georgia  Department  of  Education 

Director,  School  of 

Community 

Nutrition 

Judieth  Hunt 

Georgia  Department  of  Educahon 

Education  Grant 
Program 

Consultant 

Eugenia  Seay 

Georgia  Department  of  Education 

School  Nutrition 
Program  Manager 
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Table  B.3 

Individuals  Interviewed  in  Nebraska 


Individual 

Interviewed 

Agency 

Title 

Connie 

Stefkovich 

Shawn  Voudracek 

Alisanne  EUs 

Mary  Ann 

Brennan 

Nebraska  Department  of  Education 

Nebraska  Department  of  Education 

Nebraska  Department  of  Education 

Nebraska  Department  of  Educabon 

Administrator- 
Nutrition  Services 
Consultant- 
Nutrition  Services 
Consultant- 
Nutrition  Services 
Consultant- 
Nutrition  Services 

Table  B.4 

Individuals  Interviewed  in  New  Jersey 

Individual 

Interviewed 

Agency 

Title 

Kathy  Kuser 

Barbara  Guamieri 

Joanne  Lontz 

Jill  Niglio 

Bureau  of  Child  Nutrition  Programs,  New 
Jersey  Department  of  Agriculture 

Bureau  of  Child  Nutrition  Programs,  New 
Jersey  Department  of  Agriculture 

Bureau  of  Child  Nutrition  Programs,  New 
Jersey  Department  of  Agriculmre 

Bureau  of  Child  Nutrition  Programs,  New 
Jersey  Department  of  Agriculture 

State  Director 

Program 

Development 

Specialist  I-Public 

Program 

Development 

Specialist  1- 

Nonpublic 

Regional 

Coordinator- 

Central 

Table  B.5 

Individuals  Interviewed  in  New  York 

Individual 

Interviewed 

Agency 

Title 

Frances 

O’Donnell 

Linval  Foster 

Debbie  Favro 

Sandy  Sheedy 

Child  Nutrition  Program  Administration, 
New  York  State  Education  Department 

Child  Nutrition  Program  Administration, 
New  York  State  Education  Department 

Child  Nutrition  Program  Administration, 
New  York  State  Education  Department 

Child  Nutrition  Program  Administration, 
New  York  State  Education  Department 

Coordinator 

Associate 

Assistant 

Nutrition  Program 
Representative 

Individuals  Interviewed  in  the  Seven  States 
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Table  B.6 

Individuals  Interviewed  in  Texas 


Individual 

Interviewed 

Agency 

Title 

John  Perkins 

Debbie  Owens 

Texas  Education  Agency,  Child  Nutrition 
Programs 

Texas  Education  Agency,  Child  Nutrition 
Programs 

Director 

Program  Director 
for  Compliance 
and  Monitoring 

Table  B.7 

Individuals  Interviewed  in  Wisconsin 

Individual 

Interviewed 

Agency 

Title 

Julie  Cox 

Carol  Philipps 

Department  of  Public  Instruction,  Food 
and  Nutrition  Services 

Department  of  Public  Instruction,  Food 
and  Nutrition  Services 

Child  Nutrition 
Program 

Consultant 

Program 

Coordinator 

( 
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Non-Discrimination  Policy 

The  U.S.  Department  of  Agriculture  (USDA)  prohibits  discrimination  in  all  its  programs  and 
activities  on  the  basis  of  race,  color,  national  origin,  gender,  religion,  age,  disability,  political 
beliefs,  sexual  orientation,  and  marital  or  family  status.  (Not  all  prohibited  bases  apply  to  all 
programs.)  Persons  with  disabilities  who  require  alternative  means  for  communication  of 
program  information  (Braille,  large  print,  audiotape,  etc.)  should  contact  USDA's  TARGET 
Center  at  (202)  720-2600  (voice  and  TDD). 

To  file  a  complaint  of  discrimination,  write  USDA,  Director,  Office  of  Civil  Rights,  Room  326-W, 
Whitten  Building,  14th  and  Independence  Avenue,  SW,  Washington,  DC  20250-9410  or  call 
(202)  720-5964  (voice  and  TDD).  USDA  is  an  equal  opportunity  provider  and  employer. 


